IEL DAU GMU beyondkirkpatrick rossettelearningsymposium.gmu.edu/2010/Presentations 2010... ·...

(c)2010 Allison Rossett 1

Allison [email protected]

BEYOND KIRKPATRICKBEYOND KIRKPATRICKA FRESH LOOK AT A FRESH LOOK AT ANALYSIS AND EVALUATIONANALYSIS AND EVALUATION

© 2010 Allison Rossett

Evaluation is not working todayEvaluation is not working today

Only ¼ of respondents believe that they are “getting a solid bang” for their evaluation buckg g g

Kirkpatrick is most used approach, with Brinkerhoff second. But none of it happens much.

Five percent of training budget is spent on evaluating, mostly gathering reactionsg, y g g

Solution from the study: seek metrics that resonate for leaders

[ASTD/i4cp study, 2009]


WHERE ARE WE TODAY?WHERE ARE WE TODAY?

1. Economic pain and upheaval

44 trends affect analysis and evaluationtrends affect analysis and evaluation

2. Technology3. Evidence-based decision-making4. Workplace learning and support


Economic upheaval propels the other trends





Forty-two percent of Forty-two percent of organizations anticipate decreasing classroom learning

Seventy-two percent intend to increase intend to increase their asynchronous e-learning.

[Chief Learning Officer’s Business Intelligence study]





What What CEOsCEOs MeasureMeasureRanking Description

“of greatest concern”

1 Sustained and steady top-line growth 37.5%

2 Profit growth 36.1%

3Consistent execution of strategy by top management

33.4%

4 Speed, flexibility, adaptability to change 33.1%

5 Customer loyalty / retention 29.4%

6Stimulating innovation / creativity / enabling entrepreneurship

23.9%

Source: The Conference Board

entrepreneurship

7 Corporate reputation 22.9%

8 Speed to market 22.7%

9 [Product] Innovation 20.8%

10 Improving productivity 20.3%

94%100% Percentage of Courses Evaluated at Each of

What What WeWe MeasureMeasure–– WHEN we measureWHEN we measure

34%

13%3%

20%

40%

60%

80%Evaluated at Each of

Kirkpatrick’s Four Levels

Source: “Restructuring: Results From the ASTD Benchmarking Forum”

3%0%

Level 1 Level 2 Level 3 Level 4

Reaction Learning Behavior Results

It does not have to be this way.


The casinos LOVE data

Constant measurement against a few targets

Use of data to anticipate needs and anticipate needs and target services





Delivering support into workDelivering support into work

Coast Guard boarding officers boarding officers must know about many vessels

Unacceptable error rate; costly trainingN th Now they use a blend, a short course plus PDA to inspect and report

West Point plus ….. a blog in their dangerous “workplace”


NEW METRICSNEW METRICS

Global study in a financial services organization

Work in progress

Kirkpatrick’s model is good. Kirkpatrick’s model is good. Is it sufficient?Is it sufficient?

Level 4: Does it matter? Does it advance ?

Today, we require more

More purposes….strategy?Level 3: Are they doing it (objectives)

consistently and appropriately?+++++++++++++++++++++++++

Level 2: Can they do it (objectives)? Do they h h kill d bili i ?

More purposes….

show the skills and abilities?Level 1: Did they like the experience?

Satisfaction? Use? Repeat use?

“Allison, don’t you like Kirkpatrick’s levels?


Workplace learning/support

More purposes

Matched methods

PURPOSES. To find out….PURPOSES. To find out….1. Who our people are: their strengths, needs,

priorities, opportunities2. About the quality of the fit between what

we are delivering and what is required3. Why people perform and don’t and what

will increase alignment, transfer, performance4. How satisfied people are with learning

events and instructors, and on-demand and workplace based resources

©2010 Allison Rossett


To find out….To find out….5. If we are fulfilling promises to regulators6. If learning happened and if it transfers to

work7. If we are influencing prized strategic

outcomes8. If we are contributing to talent management,

including attraction, retention and advancement of individuals

9. The tally of all that we do and how much it is worth to the organization


To find out….To find out….

10. The adequacy of guidance to employees h bl fi d h h d so they are able to find what they need

when they need it, to self serve their development

11. If our people are engaged with their performance and developmentH i d l h 12. How committed our people are to the work and task and their willingness to contribute to networks, communities and goals

Web 2.0



Purposes in brief1. Who our people are, their strengths,

needs, priorities2. Fit between what they need and what

we deliver

3. Why they perform and don’t to tailor l ti d i fl

4. Satisfaction with what we offer, from t t d d solutions and assure influence events to on demand resources

5. Fulfillment of regulatory and compliance obligations

6. Learning happens and turns into action

7. Contributions to strategic results 8. Talent management, including attraction, retention and career advancement

9. Tally of magnitude of our efforts and 10.Guidance that enables our people to the value delivered to the organization

find what they need, to track progress, to select appropriate resources

11. Engagement with their own development and work

12.Willingness to add to community, networks, and organizational goals


You can’t tackle all the purposes.You can’t tackle all the purposes.This was a project for a mutual fund company. Two years prior they had launched an initiative to convert 45 trainers to performance consultants. Now they want to know how it’s going.

What are their purposes? No, they can’t pursue all12. No time, few resources.

1. Who our people are, their strengths needs priorities?strengths, needs, priorities?

3. Drivers and causes to assure alignment and transfer?

7. Contributions to strategic results?

It’s all about conversations!



How to find out “how it How to find out “how it waswas going”going”I limited my purposes. What methods then?◦ Who our people are, what they most need Anonymous online survey for 45 consultants seeking their assessment of skills and

knowledge associated with performance consulting Seek confidence to performknowledge associated with performance consulting. Seek confidence to perform. Interview consultants about lingering questions, and barriers what they require to

deliver on change in their roles Interview randomly selected customers. What did they handle well? What not so

well?

◦ Examine alignment by querying causes/drivers In anonymous online survey for consultants, include a question about

drivers/blockers. What would help them move forward to deliver on this new role? Interview consultants’ supervisors regarding what drives/blocks performance Interview consultants supervisors regarding what drives/blocks performance Look at the performance mgmt system. Has it changed with the new roles? See Brinkerhoff SCM

◦ Determine contributions to strategic results Interview randomly selected customers. What were they seeking when they came to

us? What business results? What indicators would signify success? Measure. Repeat requests from customers? Better framed requirements?

Copyright 2010 Allison Rossett

Some leaders are griping about the new on-boarding program….

The EVP is worried that the current program skips over how to deliver difficult messages.

Ten months ago you and your team rebuilt the program and shifted it online Now some execs

You are tasked with building something to help service reps deliver unwelcome news like online. Now some execs

are griping. What to do?unwelcome news, like that our company isn’t going to cover the claim.

Give it a goGive it a go


Workplace learning/support

More purposes

Matched methods

Use purposes to plan methodsUse purposes to plan methodsPurposes Sources Questions Indicators

To see if learning occurred

ParticipantsSupervisorsCustomersRecords

How would you handle this? (assessments, tests)

Can you ID errors?

Test scoresError rateCall backsCustomer satRecords Can you ID errors? Customer satSpeedy completion

To determine contributions to business outcomes

ExecutivesManagersStrategy docsSee Spitzer LEM

What do you expect? What indicators would satisfy? Delight?

Error rateCall backsCustomer satSpeedy completionTailored indicators

To see if we Employees Why did you join us? Recruitmentcontribute to talent management

p yPotential employeesHR, managers

y y jWhy are you departing? How do you perceive learning offering?

Retention rateEmployee satisfactionEngagement

To see if we advance careers of our people (talent mgmt)

Participants; peersSupervisors HR colleaguesPerf mgmt system

Where to from here for you? What more must you know and do? Do you know how to move forward? Resources?

Are career paths specified? Increase in promotions from within? Retention up?



Mes

sage

Mes

sage

M More data, more sources, more often

A Actionable: PLAN, REPORT, IMPROVE

R Repurposing methods, datay M

arbl

e M

y M

arbl

e M

p p g ,

B Baked in– as you build programs

L Lean, based on smaller bites of data

E Everywhere learning, support & info go

My

My


“If I h d i ht h “If I had eight hours to chop down a tree,

I’d spend six sharpening my ax.”

~~ Abraham Lincoln


[(800) 956-7739]

Useful links, I hopeUseful links, I hopeSDSU Encyclopedia of Educational Technology

http://coe.sdsu.edu/eet/

SDSU EDTEC graduate programs http://edweb.sdsu.edu/Edtec/distance/

Rossett & Schafer’s book:Job aids and performance support:Moving from Knowledge in the Classroom to knowledge everywhere

http://www.colletandschafer.com/perfsupp/index.html

Rossett’s First Things Fast book http://www.jbp.com/rossett.html

Rossett’s Beyond the Podium book http://www.pfeiffer.com/go/BTP;

A social network devoted to non-training interventions

www.pinotnet.ning.com

Pithy video introductions to Web 2.0 strategies

http://www.commoncraft.com/show

scroll downPl follow me on twitter @arossett


[http://edweb.sdsu.edu/People/ARossett/TD_Feb08_Rossett_LEGEND.pdf]

Dr. Allison Rossett, long time professor of Professor of Educational Technology at San Diego State University, is in the Training magazine Hall of Fame, CLO Advisory Board, US Army’s Distance Learning Advisory Board, and Elearning Guild’s Board too. Rossett has authored six books, including a brand new edition of her classic, First Things Fast: a handbook for performance analysis. Allison writes about and consults on learning, technology and results. You can reach Allison at [email protected] osse @ a .sdsu.edu


ASTD (2009). State of the Industry Report. 2009. ASTD.

Alvarez, K., Salas, E., & Garofano, C. M. (2004). An Integrated Model of Training Evaluation and Effectiveness. Human Resource Development Review, 3, 385.

Boudreau, J. W., & Ramstad, P. M. (2006). Talentship and HR Measurement and Analysis: From ROI to Strategic Organizational Change. HR. Human Resource Planning, 29(1), 25.g, ( ),

Brinkerhoff, R. O. (2005). The Success Case Method: A Strategic Evaluation Approach to Increasing the Value and Effect of Training. Advances in Developing Human Resources, 7(1), 86.

Holton III, E. F. (2005). Holton's Evaluation Model: New Evidence and Construct Elaborations. Advances in Developing Human Resources, 7(1), 37.

Kim, K, Bonk, C.J. & Oh, E. (September 2008). The present and future state of blended learning in workplace settings in the United State Performance Improvement blended learning in workplace settings in the United State. Performance Improvement 47(8), 5-16.

Kirkpatrick, D. (1959). Techniques for evaluating training programs. Journal of the American Society of Training Directors, 13 (3-9), 21-26.

O'Driscoll, T., & Sugrue, B. (2006). Valuing Human Capital and HRD: A Literature Review (Literature Review): IBM Almaden Services Research.

Phillips, J. (2003). Return on Investment in Training and Performance Improvement (2nd ed.). Boston, MA: Butterworth-Heinemann.

Pulchino, J. (August 2006). Usage and value of the Kirkpatrick four levels of training evaluation research report. A eport published by the Elearning Guild, www.elearningguild.com.

Rossett, A. (2009). First things fast, a handbook for performance analysis, 2nd edition. , ( ) g f , f p f y ,SF:Wiley/Pfeiffer. www.jbp.com/rossett.html

Rossett, A. (2007). Leveling the Levels. Training and Development, 61(2), 48-53.

Rossett, A. & McDonald, J. (Vol 11, 2006) Evaluating technology enhanced continuing medical education. Medical Education Online. http://www.med-ed-online.org/pdf/t0000074.pdf

Rossett, A. (March, 2010). Metrics matters. Training & Development. http://www astd org/TD/Archives/2010/Mar/1003 MetricsMatter htmhttp://www.astd.org/TD/Archives/2010/Mar/1003_MetricsMatter.htm

Russ-Eft, D., & Preskill, H. (2005). In Search of the Holy Grail: Return on Investment Evaluation in Human Resource Development. Advances in Developing Human Resources, 7(1), 71.

Spitzer, D. R. (2005). Learning Effectiveness Measurement: A New Approach for Measuring and Managing Learning to Achieve Business Results. Advances in Developing Human Resources, 7(1), 55.

IEL DAU GMU beyondkirkpatrick rossettelearningsymposium.gmu.edu/2010/Presentations 2010... ·...

Documents

Transcript of IEL DAU GMU beyondkirkpatrick rossettelearningsymposium.gmu.edu/2010/Presentations 2010... ·...