A Fresh look At meAsuring impAct - kenblanchard.com · Created by Marjorie Blanchard, PhD, ......

About the Office of the FutureCreated by Marjorie Blanchard, PhD, the Office of the Future studies and reports on emerging trends in leadership, technology, and other workplace issues.

Another primary function is to challenge the company’s status quo and act as a catalyst for change that will ensure the company’s continued vitality and success. Its findings and reports are available to clients and other organizations to assist with planning for the future and to the media for their use in keeping their readers advised of trends in the workplace.

FromtheOfficeoftheFutureofTheKenBlanchardCompanies®

A Fresh look At meAsuring impAct

Information

This document is a summary of research compiled and prepared by The Ken Blanchard Companies®, Office of the Future. It is available in PDF format. For further information, contact Lily Guthrie at

800 728-6000 or [email protected].

Proofreaders Barbara Akers Carey Nash

Production Artist Tooney Pearce Carissa Lozano

Contributions by Francisco Gomez and

Office of the Future Team

Researcher/Author Lily Guthrie

MK0667

125 State Place, Escondido, CA 92029 USA Global Headquarters 760 489-5005 • 800 728-6000 • Fax 760 489-8407

UK +44 (0) 1483 456300 Canada 905 829-3510 • 800 665-5023 Singapore +65 6775 1030 www.kenblanchard.com

a Fresh Look at measuring impact

© 2009 The Ken Blanchard Companies. All rights reserved. Do not duplicate 042909 • MK0667 �

Table of ContentsIntroduction............................................................... Page.2

The.Five.Levels.of.Evaluation.................................... Page.4

Six.Reasons.to.Evaluate.the.Impact.of.Training...... Page.5

Blanchard®.Approaches:.No.One.Size.Fits.All......... Page.7

Appendix.................................................................... Page.22

Resources................................................................... Page.23

Notes.......................................................................... Page.24

.

Kirkpatrick.Four-Level.Evaluation.Model

. Level.1—Reaction............................................. Page.4

. Level.2—Learning............................................. Page.6

. Level.3—Behavior............................................. Page.9

. Level.4—Results................................................ Page.13

Phillips.ROI.Tool

. Level.5—ROI..................................................... Page.19


� © 2009 The Ken Blanchard Companies. All rights reserved. Do not duplicate 042909 • MK0667

specific goals and objectives set for the training program with quantitative methods to measure each objective. As a result of the training, did the number of errors decrease or did the speed of service increase? They may want to administer pre- and posttraining assessments to determine if there has been a significant difference in the participants’ skills after the training compared with their ability going into the training. Based on the objectives of the training, a number cruncher will be interested in how much change they can expect to see on the job as well as how much and when the participants’ improved performance will affect the unit’s productivity.

Employee Oriented

At the opposite end of the continuum is the employee-oriented individual, who is interested in providing his or her employees with an educational experience and therefore in the quality of the training and its effect on the employees. Training is viewed as a means to an end. They are interested in whether the training program is well organized and well received. Did it motivate and inspire the participants? How did the participants react to the program? Did it offer them opportunities to discuss individual concerns and problems? The unintended

on who will be using and analyzing the evaluation data and how the data will be used.

Jane Holcomb, author of Make Training Worth Every Penny, explains that if you take all training evaluation methods and line them on a continuum from the most scientifically rigorous to the most basic methods, they can be divided into three segments. On one end of the continuum is statistical, in the center is business, and on the opposite end of the continuum is employee.

Statistical Oriented

An individual who falls in the statistical segment is a number cruncher, an analyzer, or statistically oriented. They look at degrees of cause and effect and scientific methods to control outside variables. They think in terms of facts and figures. This type of individual prefers to collect statistical data with a heavy emphasis on validity and reliability. They want to prove that the time and money spent on the training paid off and will devise ways to collect numerical data to track and monitor participants’ progress. For example, they may want

IntroductionThe steadily increasing investment in training and development over the past decade illustrates how highly committed organizations are to investing in their people. The 2008 ASTD State of the Industry report indicated that US organizations spent $134.9 billion on training and development in 2007. Nearly two-thirds was spent on the internal training function, and the remainder ($50.77 billion) was allocated to external services. Clearly, investing in employee training is recognized as a smart business investment, and more and more learning professionals are seeking viable solutions for effectively measuring the impact of training. Deciding on an appropriate evaluation process, however, varies from client to client. Blanchard® associates have discovered that clients have differing needs, wants, and objectives for evaluating training, and there is no “one-size-fits-all” solution.

For example, some clients rely on pre- and posttraining assessments; many are satisfied with getting favorable reactions to the training; and yet others expect a full-scale performance improvement analysis. As a result, not all training initiatives are evaluated using one specific method. Each situation depends

evaluation process

Rigorous Basic

Statisticaloriented

Employee oriented

Businessoriented



Introduction

ensure knowledge, skill, and attitude development.

For example, they will want to evaluate how the participants reacted to the program when the class was finished. After attending the class, did the participants absorb what they were taught? This assessment may be done in the classroom in the form of quizzes to test if the trainees are competent and confident about their newly acquired skills. This business-oriented person will want the participants to transfer what they learned back on the job. Follow-up tools such as pre- and posttraining assessments, Impact Maps, the 5 Minute Follow-Through®, and coaching can help maximize the transfer of skills.

Therefore, each individual has a different management style and orientation toward training. What is convincing, in terms of presenting evaluation data that will be used to improve employees’ performance, will vary from one client to another. There is no one-size-fits-all solution. Where the client falls on the continuum—from the most scientifically rigorous evaluation process to the most basic—will help determine the most appropriate measurement and evaluation solution. For some clients it will be streamlined and simple, and for others it will be complex and

outcomes of the program may be more significant than the program objectives, for example: sending people to a sales training program to improve sales and then discovering that the program improved their self-confidence and communication skills. An employee-oriented person is more interested in noticing if there is an improvement in the participants’ morale and performance than in collecting hard data. In their perspective, it is important that the goals and objectives of the program fit the participants’ specific needs. Receiving positive training evaluation forms and testimonials from the participants would be convincing enough for this management style.

Business Oriented

In the middle of the continuum resides the business-oriented individual. This person is practical and has a thorough understanding of the company’s present and future needs. They consider how well the program helped the department and the organization and how it affected the company’s bottom line. Evaluating if the training is producing more competent employees and improving the use of the company’s human resources are important factors. The business practitioner is interested in the participants’ reactions and also wants to

A client’s

management style

and orientation

toward training will

fall somewhere

between the two

extremes on the

evaluation process

continuum.

Knowing and

understanding

the client’s

management style is

important in order

to present them

with evaluation

data that will be

useful to improve

their employees’

performance.

sophisticated. The point is to develop an evaluation system that best matches and meets the unique needs and requirements of the client.



Level 1—ReactionMeasures customer satisfaction—how the training participants responded to a seminar or program based on a preset measurement.

A standard form is designed to quantify the participants’ feedback and participants are encouraged to write down their comments and suggestions. In a situation where participants pay for the training, their reactions may determine whether they attend again or recommend the training to others from their organization.

At Blanchard, the standard practice is to measure the reactions of the participants at the end of every workshop. Participants fill out a questionnaire, rate the training experience, and make comments. Evaluations are summarized by Professional Services, and then a report is sent to the consulting partner, sales consultant, and project manager. Evaluation data is used by Professional Services for performance review and employee development.

Did he or she like it? What was his or her reaction to the training itself and to the class?

levels. For example, a typical large organization may set the following evaluation targets:

• Level 1, 100 percent of initiatives• Level 2, 50–70 percent of initiatives• Level 3, 25–35 percent. The percentage drops because of the time and expense of conducting follow-up evaluations. • Level 4, relatively small or about 10 percent• Level 5, only the most important training initiatives

Phillips argues that whenever possible, evaluations should be an important consideration starting at the beginning of the planning process for significantly large performance improvement initiatives.

Many instructional designers develop courses using the ADDIE model (assess, design, develop, implement, evaluate) or conduct a needs analysis when developing courses, and consider Kirkpatrick’s levels of evaluation after the course is off and running. According to Kirkpatrick, a high percentage of organizations do a great job of measuring Levels 1 and 2 but few evaluate at Levels 3 and 4. The Level 5, or ROI, is becoming a standard tool by organizations evaluating for productivity improvement and gain-sharing programs as well as reward systems and safety programs.

According to Jack J. Phillips, not all training initiatives need to be evaluated at the same

The Five Levels of Evaluation

Donald L. Kirkpatrick, coauthor of Transferring Learning to Behavior, is the creator of the Kirkpatrick Four-Level Evaluation Model, which is commonly used today as the standard for training and performance evaluation. The four levels described by Kirkpatrick are Reaction, Learning, Behavior, and Results. Jack J. Phillips, author of Return on Investment in Training and Performance Improvement Programs, added Level 5, Return on Investment (ROI). The easiest and most commonly applied tools to gather evaluation data by learning professionals are Kirkpatrick’s Level 1 and Level 2.



According to

Kirkpatrick, a

high percentage

of organizations

do a great job of

measuring Levels

1 and 2 but few

evaluate at Levels 3

and 4. The Level 5,

or ROI, is becoming

a standard tool

by organizations

evaluating for

productivity

improvement

and gain-sharing

programs as well as

reward systems and

safety programs.

levels sequentially to present strong evidence that effective training led to targeted learning, which contributed to critical on-the-job behaviors that influenced the bottom line.

ownership for learning and prevent organizations from offering more strategic and rewarding courses and programs. Level 2 evaluation is appropriate for testing compliance.

4. To maximize the value of training.

It is critical to show that training is adding value and contributing to the organization’s bottom-line results. Level 3 evaluation becomes crucial for reinforcing mission-critical behaviors. Data collected through surveys, observation of behavior, and focus groups will help to cement the learning. Barriers can be uncovered to determine possible gaps preventing the trainee from applying skills to the job.

5. To align training and strategy. This involves linking organizational and individual performance with organization goals or specific business drivers. The process begins by conducting a needs or knowledge gap analysis to uncover specific expectations and the new behaviors required to accomplish those expectations. Level 3 and 4 metrics data and information are appropriate.

6. To demonstrate the value of training. Data and information is collected at each of the four

Six Reasons to Evaluate the Impact of TrainingManagement consultant Jim Kirkpatrick describes six reasons for evaluating training impact in an article entitled “The Hidden Power of Kirkpatrick’s Four Levels” in the August 2007 issue of Training and Development magazine.

1. To determine the efficacy of the program and if it should be continued.

Level 1 data collected can reveal if program material is relevant to current practices, and Level 3 data can be used to determine if people are applying what they learned.

2. To improve the program. Evaluation data can be compared with the program purpose and objectives. This works best with a feedback loop of data and information from Level 1 surveys and focus groups and from Level 2 methods. Level 3 surveys and behavioral observations can be used to determine application of skills to the job.

3. To ensure learning compliance.

Mandatory compliance is a necessary evil. However, trainers must keep training from being perceived as mandatory—that is pushing rather than pulling trainees to take a course—or it may discourage individual



Level 2—LearningMeasures the extent to which participants changed their attitudes; gained greater knowledge of concepts, principles, and techniques taught; and developed or improved skills as a result of the training program. Written tests and pre- and posttraining interviews are used by the trainer to evaluate whether the participant is competent and confident about using new skills before leaving the classroom.

When practical, a control group of people who have not yet received training can be measured against people who participated in the training to determine any learning and attitude changes. It is important to measure knowledge, skill, and/or attitudes before and after the training. Tests can be used to evaluate new knowledge and attitudes. To increase the likelihood that learning will take place, a program design may include pretraining discussions with managers and direct reports to develop clear expectations for learning and posttraining expectations on applying new skills to the job.

Success at Level 3 depends on success at Levels 1 and 2. It is unrealistic to expect a change in behavior if appropriate learning has not occurred, and it is unrealistic to expect learning to occur if steps have not been taken to create a positive learning atmosphere.

Many training courses offer Level 2 evaluation. In many cases it is considered a difficult level to measure for soft skills training. Unless the objectives of the course are behavioral, measurable objectives, this level is not effective for soft skills courses.

The most common type of Level 2 measurement at Blanchard is through skill practice in traditional (classroom) settings. Participants are provided opportunities to practice skills using real-world reinforcement scenarios that use new concepts and tools including roleplays, simulations, and case studies. Some consulting partners quiz participants to test knowledge. To develop a reliable Level 2 evaluation, course objectives need to be clear, specific, and actionable. This requires identifying learning goals for Blanchard off-the-shelf and customized products to accurately measure and evaluate the course objectives.

Do trainees have a new understanding of the ideas and methods taught? Are their skills and attitudes better? Did they learn and understand the concepts?

Six Reasons to Evaluate the Impact of Training



Understanding and

identifying a client’s

management style and

orientation toward

training will help

determine what to

measure in a training

program and how to

measure for it. For

some clients it will

be streamlined and

simple, and for others

it will be complex

and sophisticated. The

point is to develop an

evaluation system that

best fits and meets the

client’s unique needs

and requirements.

Follow-Through can help reinforce the learning by tracking and reporting real progress and behavior change as a result of the training. Follow-up surveys to participants and/or to their managers, colleagues, and/or direct reports can rate effectiveness in applying skills back on the job or whether the learning was of any value on the job altogether. These evaluators are looking for reasonable results that demonstrate value and can help the organization fine-tune its training programs.

A statistical practitioner may want to administer pre- and posttraining assessments to measure if there were significant changes or improvements in the behaviors of participants since the program, and if there was a positive link between the training and organizational and business results. They may work with the participants and managers prior to the training initiative to identify Level 4 business results (higher productivity, higher profit margin, improved customer service) and organizational results (improved morale, better communication, lower turnover) the company wants to achieve, then set up a system to track and measure progress. The statistical orientation will look for ways to automate the collection of evaluation data and track and report the short-term as well as the long-term results of training initiative.

Blanchard® Approaches: No One Size Fits All

Blanchard clients vary considerably on what is convincing evidence to them when selecting the most appropriate evaluation solution for measuring training impact. For example, a client with an employee orientation toward training may be satisfied with Level 1 measurements such as offering a questionnaire at the end of the program and asking participants to rate the session in terms of its overall value, relevance of the concepts and skills to their business challenges, and their degree of readiness to apply what they learned. Measurements such as these can provide a basis for communicating the anecdotal value of the program and feedback for improving the program delivery.

A client with a business management style, who is demanding that training not only culminate in results but also support the organization’s business objectives and strategies, may be seeking a more comprehensive solution that involves a wider range of measurements. Quizzes can measure the degree to which participants understood the principles and learned the concepts. Level 3 follow-up tools such as the 5 Minute



Blanchard’s

systematic

approach ensures

that participants

will be three to

four times better

equipped to apply

their learning and

skills and make a

business impact.

Continued on page 10

business objectives. An organization’s vision, mission, and strategic plan must be clearly linked to training objectives so that strategies, competencies, and critical tasks are in alignment and people are equipped to do their job.

5. Effective training deployment. The core message is that training is an ongoing process, not a one-time event. Blanchard’s systematic approach ensures that participants will be three to four times better equipped to apply their learning and skills and make a business impact. This success factor includes four steps to ensure that training creates tangible results for the organization.

• Setting the context. This stage ensures that training

is aligned with the vision, goals, and strategies of the organization and is supported by top management. Blanchard uses Impact Mapping tools to connect training to business results and to clarify how individual jobs, roles, responsibilities, and goals relate to the organization’s goals and strategies.

• Acquiring and applying new concepts and skills. Ideally, the training

delivery should be adapted to individuals’ learning

administering pretraining assessments; Impact Mapping; aligning roles, competencies, and key results to the organization’s goals; and connecting training to business results to develop the appropriate training strategy.

2. Following up and reinforcing. Follow-up and reinforcement after the training will cement the concepts and ensure that the people apply what they learned. Follow-up and reinforcement strategies to sustain change and ensure learning include Impact Mapping, online etools and monitoring via the 5 Minute Follow-Through, posttraining assessments, coaching, sustainability modules, connecting new behaviors and action plans to the performance management system, and adoption of key best practices such as One on One meetings.

3. Demonstrating tangible value. This step involves determining the changes and impact the client wants to achieve from the training. It must be clear how training creates change, demonstrates value, and makes a positive impact on people and performance.

4. Ensuring strategic integration. Training and learning must be a strategic and ongoing process linked to specific

Blanchard’s Five Critical Success Factors

Blanchard’s extensive research identified five key criteria for helping organizations create a sustainable approach to training that integrates numerous processes and tools to maximize and demonstrate the benefits of the training investment. The five critical success factors include:

1. Securing top management buy-in.

It is highly beneficial if management buys in to the learning process and that the training initiative aligns with the client’s strategic initiatives. Setting the context for the training involves aligning the client’s strategic objectives to the training goals, which may include




LEVEL 3—BehaviorExamines and quantifies on-the-job behavioral change. This is the most difficult to measure and probably the most important. If the trainees do not apply what they learned, the program may be considered a failure even if learning has taken place. Therefore, measuring behavioral change is necessary, not only to see if behavior has changed but also to determine the reasons why change has not occurred.

Level 3 evaluation measures the transfer of skills and attitudes that were taught in the course, to the job. It is based on the course objectives, which are tied to the performance objectives for job improvement. Level 3 requires management involvement throughout pretraining planning to identify the objectives important for emphasis and posttraining follow-up with the group to determine if the participants’ behavior changed as a result of the program. Using multiple-choice tests to measure knowledge and attitudes and either using a performance test or getting feedback on performance to measure applied skills are appropriate to see how well the participants perform after the training compared to their performance before the training.

In addition, according to Kirkpatrick, the four conditions must be met for behavioral change to occur. The person being trained must want to change, must know what to do, must know how to do it, and must work in a climate that rewards change. Allowing time after training for the behavior change to take place is important, as a trainee may not immediately apply the learning gained. Evaluation of the behavior before and after the training is critical, and a control group can be used for comparison. Surveys or interviews with the trainees, managers, and others who observe the participants’ behavior can be used.

At Blanchard, Level 3 application of new concepts and skills is measured and evaluated in a number of ways. For example, many clients use the Success Case method to conduct impact studies. Some clients conduct surveys to collect data on the application of the learning. The 5 Minute Follow-Through is one way to track and monitor the application of one to two specific goals. Some clients conduct pre- and posttraining assessments using the Leader Action Profile. Follow-up and reinforcement strategies to improve retention of skills and knowledge include executive and individual coaching, and sustainability modules.

Have the individuals’ behaviors become different as a result of training? Can they use the new skills on the job? Did training teach how to adapt new skills to the job, unique problems, and situations? Are these changes observable?



�0 © 2009 The Ken Blanchard Companies. All rights reserved. Do not duplicate 042909 • MK0667

So when evaluating

training impact,

we are also

evaluating an

organization’s

performance

management

system.


Continued from page 8

solution. In contrast, other HR clients already have a seat at the executive level and are intimately involved with the manpower development needed to grow the company.

Success Case Method

The Success Case Method (SCM) developed by Robert O. Brinkerhoff, coauthor The Success Case Method, is a simple and proven way for evaluating how well a new organizational initiative or training program is working. By applying SCM, learning professionals can gain access to useful and accurate data and information on whether a program is working and how the program can be improved.

Brinkerhoff emphasizes that all training produces predictable or reliable results. For example, after a training program, some people immediately put their training to use and generate valuable results, such as increased sales or improved quality. There is also a percentage of people who do not apply the training and perhaps never will. In the middle are participants who may try something new on the job but regress back to their old behaviors. A quantitative approach to measuring results may yield a low score and underestimate the impact of the training. Therefore, the best solution is to measure and analyze the training results of the early adopters. If the training

• Applying new skills on the job.

In order for individuals to apply their new skills, organizations must spend 10 times the energy in the follow-up and reinforcement of training than they did in the delivery. Effective processes for Levels 3 and 4 may include coaching and the 5 Minute Follow-Through (a Web-assisted goal setting, tracking, and monitoring system). Individuals also need to be held accountable for using what they have learned. This is often achieved through integration of new skills and best practices with the organization’s performance management system.

Clients’ needs and expectations vary as some are learning professionals while others are top-level managers and leaders. Many learning professionals have little control over strategic organizational direction. In some organizations, the HR function receives little direct support from top management and is not considered a strategic partner in achieving organizational results. These HR departments may be tasked with ensuring that training events are held and that Level 1 and 2 evaluations are collected. As the relationship between Blanchard and the client develops, a window of opportunity often opens that captures the attention of senior management and results in a broader, more integrated training

preferences (visual learners, kinesthetic learners, auditory learners) as well as offer a blend of delivery methods (classroom, online, self-study, blended learning). The learning should be divided into manageable segments so that people acquire, retain, and apply new concepts more easily. Finally the appropriate number of individuals should be trained to ensure support and sustainability in the culture and create the critical mass necessary to support change.

• Practicing new skills and behaviors.

To sustain the learning, participants must practice what they have learned by interacting with the material through exercises, roleplays, games, simulations, and real-life experiences. They also need opportunities to practice new skills and behaviors and receive feedback.


© 2009 The Ken Blanchard Companies. All rights reserved. Do not duplicate 042909 • MK0667 ��


Performance consulting requires business knowledge, organizational development skills, and knowledge of industrial and organizational psychology, which not every learning professional can or wants to assume in their role. Performance consulting may involve working collaboratively with people at all levels of the organization to gain a deep understanding of their strategic targets, how they expect to execute the strategies and apply those skills and knowledge to drive expected performance, how the organizational culture inside and outside the company operates, and how to best equip people with skills that most effectively influence performance.

A shift from event-based, just-in-case training to a more performance-based, just-in-time approach moved Blanchard to performance consulting as a way to help clients link learning and performance to business goals. A number of senior Blanchard consulting partners and sales consultants are knowledgeable and trained in using this approach with clients and comfortable talking with clients about return on investment related to costs of employee turnover, employee retention, and so forth. Many consulting partners who use this approach measure the success of training initiatives through internal client satisfaction surveys, the client’s use of Blanchard resources

with anecdotal evidence that validated ROI impact. According to the data, the level of business impact from the training investment was significant. For example, as a result of SLII®

and the associated coaching, the working relationships among staff members improved dramatically. One store’s performance improved from “unacceptable” in many areas to “highly acceptable”; another store’s revenue increased from $8,000 per week to $14,000 per week; the level of performance of a new general manager accelerated; and an effective manager decided to stay with the company. The business impact was measured in terms of its economic value (Level 5), which revealed an impressive impact per person trained in excess of $30,000.

Performance Consulting

A performance consultant is a change agent who focuses primarily on what people do and then considers what it takes in terms of skills, knowledge, and a range of workplace resources to improve performance. Training is viewed as a means to an end, not the solution. Performance consultants form and build strong relationships with clients over time to understand and learn about the many aspects of the client’s business, such as the products and services, desired results, and workforce performance issues.

works and was valuable for some of the people, then the next step is to find out how to help more people use it. Successful results from peers can be integrated into the training program for added inspiration.

Brinkerhoff explains that training alone is never the sole cause of its success or failure; there are other variables that come into play, such as management support or preparation for learning that can enhance or impede its effects. So when evaluating training impact, we are also evaluating an organization’s performance management system. Rather than performing Level 3 surveys, which could produce discouraging results, we use evaluations and follow-ups to uncover the critical performance system factors that either helped or hindered the performance results. Based on this knowledge and data, we can recommend improvements to a training program or suggest related training initiatives.

To illustrate, Blanchard and a key client partnered with Brinkerhoff and the Learning Alliance to conduct an impact study applying the Success Case method. The purpose of the study was to uncover where the application of learnings from a training and coaching initiative led to the achievement of one or more leadership and/or business outcomes. The study yielded reliable results at Level 4 along


�� © 2009 The Ken Blanchard Companies. All rights reserved. Do not duplicate 042909 • MK0667

The client’s

administrative

overhead was

reduced

dramatically with

a cost savings of

$1.5 million.



2. Four, two-day site visits (at a minimum) are scheduled within a year. Each organization has the opportunity to host one of these site visits and be the recipient of a focused exploration.

3. Prior to the site visit, the host CEO, with his or her leadership team, defines the issues he or she would like the Peer Partners to address. The issues are then reviewed by the Peer Partners to agree on which issues to address.

4. The first day of the site visit, the visiting Peer Partners interview people, observe operations, hold focus groups, and use other methods to explore the agreed-upon issues. After conferring with each other, they meet with the host and share their observations.

5. On the second day, the group shares their findings with the leadership team. Action planning is done by the leadership team while the Peer Partners act as process consultants; measures are put in place to track progress toward the goals set.

6. Other activities include a report created by the Peer Partners according to the host’s specifications and a follow-up debrief meeting on refining the consultation process.

the country and serving different populations, the group shared similar beliefs and values and discovered they could help one another accomplish their individual objectives for reducing costs, increasing productivity and efficiency, and improving the business position of their respective organizations. Together they designed a structured process to encourage and support one another and act as each other’s consultants in making needed changes in their respective organizations.

Blanchard’s Building High Performing Teams® (BHPT) program provided a common language and framework that taught them and their top management teams how to work more effectively and collaboratively within a team. The process begins at the CEO level and eventually cascades to other levels of management. For example, the CFOs created a similar process with their counterparts in their other sister organizations. The model is also being modified to work within organizations and not just across organizations. The Peer Collaboration process follows seven steps:

1. Four Peer Partners make a mutual commitment to learn from one another’s experiences in a manner designed to serve the needs of each participating organization.

(e.g., client-Blanchard team) on client-sponsored projects, and the client not having difficulty in funding those Blanchard resources.

A Blanchard consulting partner who uses this style of performance consulting extensively with clients commented, “It really comes down to how much value the client organization has for the work you are doing and how willing they are to allocate dollars to let you continue to do the work.” Although not all Blanchard consulting partners and salespeople are trained in performance consulting, all of them are familiar with our offerings and are helpful in matching interested clients with these skilled resource people.

Peer Collaboration Process

The Peer Collaboration process was the brainchild of four CEOs from noncompeting behavioral healthcare institutions. Despite coming from different parts of



Level 4—ResultsMeasures the bottom line and looks at the final results of the training from the organization’s perspective. It is critical to consider the cost of the training program versus its specific benefits for the organization. Returns from training efforts can include less employee turnover, improved quantity of work, improved quality, reduction of waste, reduction of time, increased sales, reduction in costs, increased ROI, and increased profits.

A control group can be used to compare the business results of a “trained” versus untrained unit or organization (for example, two stores can be compared). Participants need time to demonstrate improved results. Kirkpatrick argues that it may not be feasible to absolutely prove the positive results of the training, since the results being measured, such as profits or turnover, are affected by so many other variables. Kirkpatrick suggests that Level 4 evaluation be done only on those programs considered most important or most expensive and that ROI measurement should only be attempted on about five percent of an organization’s programs.

To measure and evaluate business impact, Blanchard primarily relies on collecting data on performance metrics from the client through impact studies. For example, Impact Mapping in SLII® is designed to evaluate Level 4 results. Most clients, however, are not ready for or are unwilling to invest in Level 4 evaluation as it requires dedicated time and additional resources; however, they are persuaded about the impact of SLII® through anecdotes that filter up more quickly with some clients they are interested in.

Are profits up? Has quality improved? Are employees happier and more productive? Has turnover decreased? How does training affect the organization’s bottom line? Has customer service improved and have customer complaints decreased?






responsibility, cooperation and joint decision making with key stakeholders, such as managers, learning professionals, trainees, and others, are critical for developing a successful training initiative.

A productive partnership requires stakeholders to understand the concerns, problems, and opportunities of the others. First, managers can build transfer of learning into performance standards to ensure an employee receives the appropriate training and to support the application of the learning when the trainee returns to the job. Collecting baseline performance data can help to identify transfer problems and to measure the impact. Conducting a needs or knowledge gap analysis can provide data to determine the training required. Second, learning professionals can examine the business needs to determine whether the HR strategy is linked to the overall business strategy and to determine the appropriate metrics and measures to adopt. Every function within a company will require different data, but the key to determining the impact will be the choice of an evaluation method that addresses the learners’ behavior as a result of the training, the training’s impact on the business, and the return on the training investment.

In addition to bottom-line results, the four organizations realized a number of significant changes such as improved communication across departments, improved employee morale, increased productivity through the ownership and empowerment of team members, and decreased employee turnover.

Peer Collaboration illustrates a cost-effective approach to obtaining industry expert consultation on issues particular to the client’s organization and industry. Not only do peers of a similar level within the partnering organizations have the benefit of one another’s expertise but also the process can also be used within departments of the same organization. The model can be replicated in other industries. This approach goes beyond training; it is really about culture change. The organizations learned to work collaboratively in a competitive field; everyone benefited.

Customized Design and Delivery

Mary L. Broad, author of Beyond Transfer of Training, explains that training efforts must be supplemented by significant support from key stakeholders—before, during, and after a training initiative—to ensure that the desired performance is transferred to the job. Shared

7. Subsequent meetings are conducted to check on the progress toward goals and on other specific issues.

The Peer Collaboration process proved to be extremely beneficial, resulting in substantial savings in all four peer organizations. To illustrate the impact, the organization’s administrative overhead was reduced dramatically with a cost savings of $1.5 million. Direct staffing costs were reduced due to reorganization when converting to teams.

Liability risk decreased due to a reduction in staff and the outsourcing of specific services. Supply expenses were reduced by approximately $150,000 due to reduced staff and more efficient buying procedures. Because of more sophisticated team goal setting and tracking, the organizational budget grew from $23.5M in 1999 to $28.3M in 2003. Moreover, the company grew $5M in revenue in five years.

Other partnering organizations showed impressive results as well. For example, one healthcare provider reported that two departments increased productivity dramatically. Revenues increased from $9.6M in 2002 to $16.6M in 2005. During the same time, administrative charges dropped from 14.3 percent of total budget to 8.9 percent.



Peer Collaboration

illustrates a cost-

effective approach

to obtaining industry

expert consultation

on issues particular

to the client’s

organization and

industry.


training program to develop new leaders. The core concepts and skills in the program were developed based on feedback during internal interviews. The program was built to address leadership training and development needs based on senior leaders’ expectations, leadership competencies, and individual job accountabilities. The process involved an all-encompassing approach that integrated self-awareness, workshops, structured experiential job activities, action learning project, and peer coaching and feedback. The curriculum followed a four-step process: an introduction to leadership, leadership essentials, leading people, and leading business. The company built in a measurement system, that included using 360-degree pre- and posttraining assessments, Kirkpatrick’s four levels of evaluation,

with valuable feedback to quantify the overall training impact and the bottom-line value of the training initiative. Trainees and managers were asked to estimate the monetary value that the training provided the organization such as cost savings, retention value, the value of time that increased productivity or profit, increased employee engagement, and/or mastery of skills. As a result of using this process, the client was able to evaluate the results of the initial course (Level 1). Participants reacted favorably to the training and encouraged the organization to continue this learning initiative. The client measured the influence of SLII® (Levels 2 and 3). Ninety-five percent of the respondents felt the training was valuable, and 46 percent had already seen concrete results. More than 90 percent felt confident in using the leadership styles, and 73 percent said that the new skills improved the quality of communication with team members. Responses for measuring the value of the program (Levels 4 and 5) were positive. Results included instances of accelerated promotions, increased revenue, and cost savings. The process was repeated for SSL and SFL training.

Another example that illustrates a unique design involved a partnership between Blanchard and a financial institute. The client developed a nine-month

Many instructional designers use the ADDIE model (assess, design, develop, implement, evaluate) to determine the training required by the organization or individual including outcomes, objectives, skills, knowledge, mixes of training methods and media, and evaluation of results.

Blanchard’s work with a client in the healthcare services industry illustrates a comprehensive learning blueprint designed to maximize the transfer of learning and which embedded communications, measurement, and reinforcement. The firm’s blueprint included measurement and evaluation surveys to assess the effectiveness of the SLII® training initiatives and to make necessary adjustments to maintain a high quality program. Conducting timely evaluations after the training and getting feedback from trainees and their managers on how well direct reports applied newly acquired knowledge and skills back on the job provided valuable information. Surveys included feedback on how the application of the training led to business outcomes, such as retaining a high potential team member, helping someone or a team achieve important business goals, and helping a marginal performing team member become a high performing team member.

Ongoing surveys and evaluations provided the client




transferring skills and knowledge to performance in the work setting. For example, both management teams experienced dramatic improvements in overall effectiveness. At the business level, the organization met and exceeded its targeted goals for the year. Clearly, the combination of assessment, training, and performance coaching is a powerful formula for success.

Client Impact Studies

Client impact studies conducted by Blanchard to demonstrate organizational impact have been encouraging and well-used. These studies have challenged the belief that returns on training investments are not always tangible or easily expressed in dollars. Clients who have invested in this third-party effort have been highly satisfied with the results of both the training and evaluation method.

Typically, Blanchard partners with third-party vendors to work with clients to measure training impact. A comprehensive impact study ranges from $10,000 to $100,000, with the client paying for the evaluation process. Many clients are interested in doing what other clients have done to prove the impact of training, but do not necessarily want to spend the money, time, and effort to do it all themselves. Therefore, they are comfortable leaving the work to outside experts.

Analysis, PERFORM Rating Scale, Team Leader Action Profile for General Manager, and individual and group interviews. Blanchard presented the final results and recommendations, which included SLII® and BHPT training for 33 senior level managers and coaching for 8–12 months, and SLII® training for 59 supervisors. The training was well received and provided a common framework for helping teams understand their development process and a starting point for improving productivity and employee morale in the organization.

A key element of Blanchard’s solution was having a performance coach onsite during and after the training. The purpose was to ensure the management teams achieved their vision, objectives, and goals. The coach helped the teams and individuals adopt and use effective management practices, based on the needs identified and concepts and skills learned in the training sessions, as well as provide ideas and learning tools to improve their performance. From the client’s perspective, the performance coach was a clear differentiator for Blanchard, when compared to other training and consulting firms. This translated directly into results and outcomes. It produced tangible, added value to the client. Performance coaching provided Level 3 evaluation for

promotions, the percentage of internal versus external hires, program completion, and ROI. For example, Level 3 evaluation included solving a real-life business problem using the company’s operational excellence methodology. Working within the context of projects, teams of participants had an opportunity to apply their leadership knowledge and skills to issues such as communication, listening, coaching, peer influence, problem solving, teamwork, networking, conflict management, and strategic thinking. Building a measurement system into the program enabled the client to gauge success at each stage, thus ensuring the program was meeting its intended goals. The initiative was highly successful and delivered impressive results.

Performance Coaching

A Latin American-based oil firm and Blanchard partnered to create a training initiative that would attain higher levels of implementation of leadership practices and lead to improved business results. The plan was to improve the performance of two critical senior management teams in the company. They began with a needs assessment to determine potential gaps and set a baseline for the project. Instruments included the Team Charter Checklist, Team Development Stage



A key element of

Blanchard’s solution

was having a

performance coach

onsite during and

after the training.

From the client’s

perspective, the

performance

coach was a clear

differentiator for

Blanchard, when

compared to

other training and

consulting firms.


• Easy terminology for One on Ones.

• Improved my work environment with managers and coworkers.

• Will apply to my current transition to better align with manager; will apply model to all my interfaces.

• One of the best sessions I’ve been in.

In 2006, Blanchard devoted an entire Client Summit to training impact. The title was

as powerful as ROI metrics and data, especially to the group of people whom Jane Holcomb described as having an employee orientation toward training. Blanchard clients are excited to share their ideas, results, and impact to others at annual Client Summits. For example, in 2004, a key global client shared how they used SLII® with their 2,000-person sales force in North America. Blanchard and the client teamed to create a blended learning solution that combined training and etools to dramatically improve the communication and collaboration of the company’s sales leadership and teams to support future growth and expansion. The client selected SLII® because it is easy to learn, works in all cultures, is simple to deploy, is research-based—and because it works. The training initiative produced the highest evaluation scores worldwide with an average of 4.8 (5.0 scale). Internal leadership scores (using a 360-degree leadership assessment) improved within one to two years of training. Senior sales managers commented:

• Easy to practice.

• Even better then second time around.

• Allowed me to reflect on my strengths and improve my leadership opportunities.

• Will take my people through the theory and make it live.

One of the most extensive impact studies conducted was a Blanchard partnership with a global multinational client. Productivity increases as a result of SLII® training were measured using an the company’s proprietary assessment, which measures specific leadership competencies, in addition to specific SLII® skills. The research showed that direct reports attributed 75 percent of their improvements to better leadership and 36 percent of participants attributed their own leadership improvement to SLII®. The research indicated that Situational Leaders helped increase their direct reports’ productivity by 10 percent. Leadership ROI was calculated over a six-month period and assumed that a sales specialist (direct report) contributed $50,000 in revenue:

• A 10% increase yields $5,000 (benefit of $5,000).

• The investment in SLII® per participant is $2,500 (cost of $2,500).

• Thus, return on the investment In training is 200%.

Anecdotal Evidence

Blanchard training initiatives have yielded thousands of anecdotes from Blanchard clients about the impact of Blanchard training programs. Anecdotal evidence can be



The LAP provides

leaders with

feedback on

the frequency

with which

the leader uses

specific Directive

and Supportive

Behaviors; gaps

between a leader’s

self-perception and

the perceptions of

those they lead …



The LBAII® evaluates the ability of a leader to diagnose development level and choose the appropriate leadership style. The Leader Behavior Analysis II analyzes self and others’ perceptions of the participant’s leadership style, flexibility, and their effectiveness in matching the appropriate leadership style to a particular situation. The results from this questionnaire not only provide participants with valuable feedback on their leadership style but also build a foundation for understanding the concepts of Situational Leadership® II.

3. Team Performance Assessment (TPA). This assessment focuses on team leadership. It gives feedback from the participants’ team members regarding the ways they are functioning as a high performing team and the areas of opportunity for further development as a team. It is based on the PERFORM Model and provides a snapshot of a team’s strengths and gaps in modeling the seven characteristics common to all high performing teams: Purpose and values, Empowerment, Relationships and communication, Flexibility, Optimal productivity, Recognition and appreciation, and Morale. The action plan assesses the team’s productivity and

assessment business has been growing steadily due to a high volume of clients requesting assessments. We offer the following, among others:

1. Leader Action Profile (LAP). The LAP provides leaders with feedback on the frequency with which the leader uses specific Directive and Supportive Behaviors; gaps between a leader’s self-perception and the perceptions of those they lead; respondents’ levels of satisfaction; and a leader’s flexibility in using a variety of leadership styles with respondents (Level 1). For Levels 2 and 3, individuals can create a strategic plan to develop specific leadership skills and conduct a posttraining assessment to track measureable results of change by identifying key areas for improvement and setting up and tracking specific goals to change behaviors. In addition, the LAP is an effective tool for measuring Level 3 training results through increased use and effectiveness of SLII® throughout the organization. Currently, pretraining assessments outnumber posttraining assessments, suggesting that some clients are not measuring training results.

2. Leader Behavior Analysis II® (LBAII®).

Retention, Results, and Return on Investment. Client presenters included Pfizer, Banta Catalog Group, Wachovia Corporation, Toyota Motor Sales, and others.

Blanchard Assessments

A best practice is to include pre- and posttraining assessments in the training design to accurately gauge progress. Quantitative pre- and posttraining assessment evidence can build the business case for training and demonstrate how training will help the unit or organization attain its goals.

In recent years, the Blanchard



Level 5—ROI Compares the program’s monetary benefits to the program’s costs (usually expressed in a percentage). Level 4 impact data must be converted to monetary value and include the costs of the program to calculate ROI. Phillips’ ROI model provides a systematic approach to develop ROI calculations. To show that a program improved productivity, enhanced quality, reduced employee turnover, decreased absenteeism, or increased customer satisfaction, data is converted to monetary units so that the benefits can be compared to costs, which in turn leads to an ROI calculation. Collecting hard data is relatively easy; however, for soft data items such as customer satisfaction, employee turnover, employee absenteeism, and job satisfaction, the process is more difficult. However, there are techniques available to make these conversions reasonably accurate.

Level 5 evaluations, ROI, are the Level 4 results translated into monetary values. The standard formula for calculating ROI is

ROI percent = Program benefits – Program costs Program costs

The results are multiplied by 100 to show the ROI as a percentage.

ROI percent = $352,000 multiplied by 100 = 154% $229,000

Note: This implies that for each $1 invested in the program, there are returns of approximately $1.50 in net benefits after costs are covered.

At Blanchard, Level 5 ROI is data collected through client-paid impact studies.



�0 © 2009 The Ken Blanchard Companies. All rights reserved. Do not duplicate 042909 • MK0667

PLA is a reliable

and valid measure

of leadership

behavior critical to

job performance

and employee

commitment.

The online test

measures behavior

through self-report

and 360-degree

feedback.



performing organization. High performing organizations are organizations that over time continue to demonstrate outstanding results with the highest levels of morale and commitment to the success of the organization. They have developed enviable reputations as being an employer of choice, a provider of choice, and an investment of choice. The HPO SCORES Model provides the framework for the Higher Performing Organizations Profile. The six critical elements of high performing organizations are represented by the HPO SCORES acronym: Shared information and open communication; Compelling vision; Ongoing learning; Relentless focus on customer results; Energizing systems and structures; and Shared power and high involvement. The information provided by HPO SCORES can be used by leaders to identify initiatives and programs that will help the organization become a high performing organization and to measure impact of initiatives and programs.

8. Gung Ho! Quotient. The Gung Ho!® Quotient measures employee perceptions of the organization’s culture against Gung Ho! strengths of meaningful work, control of the goal, and cheering

posttraining assessment can be conducted to measure change in behavior.

5. Preferred Leader Assessment (PLA). PLA is a reliable and valid measure of leadership behavior critical to job performance and employee commitment. The online test measures behavior through self-report and 360-degree feedback. Scores are based on a normative sample. A learning program tied to the assessment provides prescriptive actions leaders can take to improve in each of the three core areas of effective leadership.

6. The DISC Profile. The DISC is a tool to help improve work effectiveness. It provides a comprehensive perspective on how an individual interacts with others in everyday situations. The assessment enables individuals to understand their personal behavioral preferences in order to enhance communication, achieve tangible results, and work more effectively with people who have different strengths.

7. High Performing Organizations (HPO) SCORES®. This instrument identifies an organization’s strengths and opportunities across six common characteristics of a high

morale, determines the team’s stage of development, and identifies the team’s needs. It allows the leader or the team to create an action plan based on strengths and gaps.

4. Empowerment Development Gauge and Evaluation (EDGE). The EDGE is a survey and a tool for seeing if there is a match between managers and employees regarding the attitudes and behaviors needed to support the concepts of Self Situation Leadership. The instrument can help bridge the gap between knowledge of Situational Self Leadership and its application. A




dramatically when training is combined with coaching. At Level 3 the true application of new skills and behaviors begins to gain traction, since the focus is on helping participants take

5. guide measurement and evaluation.

Impact Maps are an effective tool for measuring Level 2 and Level 3 training impact. Blanchard research shows that the effective use of Impact Maps linking training to individual skills to jobs and then to organization goals increases the use of skills learned from about 20 percent to about 80 percent, especially if coaching is incorporated into the initiative.

The 5 Minute Follow-Through® (5MFT)

This easy-to-use Web-based tool automates a follow-up to training. Learners share learning objectives with their manager, receive reminders of goals, and are asked to provide progress updates. They also consult with the Blanchard One Minute Mentor™, receive ongoing feedback and support from their manager, and have access to an online learning community. The 5MFT is an invaluable tool for training managers to capture and report real progress and behavior change as a result of training for Level 3 and Level 4 evaluation.

Coaching to Support Learning

Coaching is an impressively effective sustainability strategy for maximizing skills that are transferred to the job. Research shows that learning, productivity, and morale improve

each other on. It compares leader’s perceptions with employee perceptions and fosters action planning for developing a Gung Ho! culture by analyzing the gaps between perceptions and best practices.

Impact Mapping

Impact Maps are visual representations of the links between an individual’s role and responsibilities and how the capabilities (skills and knowledge) for the role influence key business results of the organization. This mapping approach ensures a strategic focus and an achievable scope for training, leading to greater employee involvement, motivation, and satisfaction. Impact Maps yield higher success rates for the transfer of training, i.e., linking training outcomes to business goals and to individual performance improvement objectives. The process provides the creation of individualized learning-performance paths that can be jointly managed by learners and their supervisors.

Impact Maps are used to: 1. analyze the business impact

that is needed from the learning initiative;

2. identify the learning and performance requirements;

3. clarify the expectations for learning results;

4. communicate learning plans and goals; and

The six critical

elements of high

performing

organizations are

represented by

the HPO SCORES

acronym: Shared

information

and open

communication;

Compelling vision;

Ongoing learning;

Relentless focus on

customer results;

Energizing systems

and structures; and

Shared power and

high involvement.





of the content and explore new skills in a confidential environment.

Through Blanchard’s proprietary Coaching Management System, participants not only receive expert coaching but also have the opportunity between sessions to practice and sharpen new skills online. This management system can also identify organizational barriers to on-the-job use of training in a way that ensures the confidentiality of participant comments.

what they have learned thus far and apply it to the job.

Key support elements include follow-up telephone coaching sessions with a Blanchard certified business coach to help participants move from learning to taking action. One-to-one conversations with a coach will help deepen their understanding

Learners share

learning objectives

with their manager,

receive reminders

of goals, and are

asked to provide

progress updates.

They also consult

with the Blanchard

One Minute

Mentor™, receive

ongoing feedback

and support from

their manager, and

have access to an

online learning

community.


APPENDIX

Research and Critical Questions

This paper examines a variety of approaches that The Ken Blanchard Companies® offers to clients to measure and evaluate training effectiveness and business impact, such as Success Case method, performance consulting, peer collaboration, impact studies, assessments, and anecdotal evidence. In addition, this paper explores opportunities for expanding Blanchard’s capabilities in order to better provide clients with multiple solutions for measuring impact including recommendations in this area to move forward.

This paper is based on extensive literature research including books, publications, and magazines by measurement and evaluation experts, interviews with Blanchard consulting partners and sales consultants, and an analysis of client impact studies, stand-alone studies, How We Work series, and Client Spotlights to determine best practices, opportunities, and recommendations.



.ResourcesAchieveGlobal. Leadership Development. 2007. Online. April 1, 2008. Available: http://achieveglobal.com/solutions/leadership

Atkinson, Tom. Measuring Up: Find out if your leadership development programs are paying off. 2008. The Forum Corporation. April 1, 2008. Available: http://www.forum.com

Brinkerhoff, Robert O. The Success Case Method: Find Out Quickly What’s Working and What’s Not. (San Francisco, CA: Berrett-Koehler Publishers, Inc., 2003.)

Broad, Mary L. Beyond Transfer of Training: Engaging Systems to Improve Performance. (San Francisco, CA: John Wiley & Sons, Inc., 2005.)

Broad, Mary L., and John W. Newstrom. Transfer of Training: Action-Packed Strategies to Ensure High Payoff from Training Investments. (Redding, MA: Perseus Publishing, 1992.)

Development Dimensions International. Leadership Research Summary. 2008. Online. April 1, 2008. Available: http://www.ddiworld.com

FranklinCovey. 360-Degree Profiles. 2008. Online. April 1, 2008. Available: http://www.franklincovey.com/southasia.com/PDT/curriculum/profiling/profiling%20overview.pdf

Holcomb, Jane. Make Training Worth Every Penny: On-Target Evaluation. (San Diego, CA: Pfeiffer & Company, 1994.)

Kirkpatrick, Donald L., and James D. Kirkpatrick. Transferring Learning to Behavior: Using the Four Levels to Improve Performance. (San Francisco, CA: Berrett-Koehler Publishers, Inc., 2005.)

Kirkpatrick, Jim. 2007. “The Hidden Power of Kirkpatrick’s Four Levels.” Training and Development. Vol. 61, No. 8: 34–37.

Paradise, Andrew. State of the Industry Report 2007. 2007. The American Society for Training & Development. ISBN-10: 1-56286-511-0

Phillips, Jack J. Return on Investment in Training and Performance Improvement Programs. 2nd Edition. (Burlington, MA: Butterworth-Heinemann, 2003.)

Phillips, Jack J. The Value of Human Capital: A Micro-Level Approach. 2005. Online. April 1, 2008. Available: http:/www.clomedia/content/anmviewer.asp?a=1097

Thompsen, Joyce, and Linda Moran. 2007. “Driving Organizational Effectiveness: Performance Consulting at Work.” Chief Learning Officer magazine. Vol. 6, No. 12: 22–25.

Wihelm, Warren. 2007. “Does Senior Leadership Buy ROI for Learning, Part 1.” Chief Learning Officer magazine. Vol. 6, No. 12: 86–88.

Wilhelm, Warren. 2008. “Does Senior Leadership Buy ROI for Learning, Part 2.” Chief Learning Officer magazine. Vol. 7, No. 1: 48–50.

Wilson Learning Worldwide. Human Performance Improvement: Connecting People and Performance to Business Strategy. 2003. Online. April 1, 2008. Available http://www.wilsonlearning.com



NOTES

125 state place, escondido, ca 92029 usaGlobal Headquarters 760 489-5005 • 800 728-6000 • Fax 760 489-8407

UK +44 (0) 1483 456300 Canada 905 568-2678 • 800 665-5023www.kenblanchard.com

A Fresh look At meAsuring impAct - kenblanchard.com · Created by Marjorie Blanchard, PhD, ......

Documents

Transcript of A Fresh look At meAsuring impAct - kenblanchard.com · Created by Marjorie Blanchard, PhD, ......