Are we fulfilling our promise
-
Upload
charles-rei -
Category
Documents
-
view
1.616 -
download
0
description
Transcript of Are we fulfilling our promise
![Page 1: Are we fulfilling our promise](https://reader034.fdocuments.net/reader034/viewer/2022050722/54990eb6b4795962238b45a5/html5/thumbnails/1.jpg)
Are we fulfilling our promise?Assessing and reporting training quality
IATEFL BESIG Annual Conference
Stuttgart, Germany
November 16, 2012
Charles Rei
![Page 2: Are we fulfilling our promise](https://reader034.fdocuments.net/reader034/viewer/2022050722/54990eb6b4795962238b45a5/html5/thumbnails/2.jpg)
Learning objectives
By the end of this session we will be able to...
1. Identify the benefits of comprehensive training assessment.
2. Identify and assess the four levels of training evaluation using quantitative and qualtative metrics.
3. Identify external factors which affect assessment.
4. Use surveys, performance-based assessment, and client contact to evaluate training.
5. Identify the main elements and organization of a comprehensive training evaluation report.
![Page 3: Are we fulfilling our promise](https://reader034.fdocuments.net/reader034/viewer/2022050722/54990eb6b4795962238b45a5/html5/thumbnails/3.jpg)
TrainingPerks
• A gift to the employees
Education
• Knowledge-based
• Standardized
Goals
• Performance-based
• Personal development
![Page 4: Are we fulfilling our promise](https://reader034.fdocuments.net/reader034/viewer/2022050722/54990eb6b4795962238b45a5/html5/thumbnails/4.jpg)
Current situation• Surveys / Feedback forms• Observation• Interviews• Testing• Self-assessment• Can do statements assessment• Portfolios and projects• Regular reports• Status meetings• Cancellations and renewals• None whatsoever
Formative vs. Summative
![Page 5: Are we fulfilling our promise](https://reader034.fdocuments.net/reader034/viewer/2022050722/54990eb6b4795962238b45a5/html5/thumbnails/5.jpg)
Benefits of comprehensive assessment
• Learner accountability• Their dedication matters
• Organizational / professional development• Strengths and areas for improvement• Quality processes• Sharing best practice
• Added / demonstrated value• Client communications and relationships• Value-based pricing vs. Cost-plus pricing• Targeted sales leads (added services, target market)
![Page 6: Are we fulfilling our promise](https://reader034.fdocuments.net/reader034/viewer/2022050722/54990eb6b4795962238b45a5/html5/thumbnails/6.jpg)
THE FOUR LEVELSAssessment Criteria
![Page 7: Are we fulfilling our promise](https://reader034.fdocuments.net/reader034/viewer/2022050722/54990eb6b4795962238b45a5/html5/thumbnails/7.jpg)
Kirkpatrick’s 4 Levels - Reaction
• Enjoyment?• Priority?• Useful?• Satisfied?• Attendance• Hours• Post-lesson contact
![Page 8: Are we fulfilling our promise](https://reader034.fdocuments.net/reader034/viewer/2022050722/54990eb6b4795962238b45a5/html5/thumbnails/8.jpg)
Kirkpatrick’s 4 Levels - Learning
• Remember?• What did you learn?• Project performance• Testing• Performance-based rubrics
• Trainer / Self assessment
![Page 9: Are we fulfilling our promise](https://reader034.fdocuments.net/reader034/viewer/2022050722/54990eb6b4795962238b45a5/html5/thumbnails/9.jpg)
Kirkpatrick’s 4 Levels - Behavior
• Did you apply the lessons?
• Pre-task reflection• Models and templates• Authentic materials
![Page 10: Are we fulfilling our promise](https://reader034.fdocuments.net/reader034/viewer/2022050722/54990eb6b4795962238b45a5/html5/thumbnails/10.jpg)
Kirkpatrick’s 4 Levels - Results
• Productivity measurements
• Better relationships• Reduced stress• Misunderstanding and conflicts
• Performance appraisals
![Page 11: Are we fulfilling our promise](https://reader034.fdocuments.net/reader034/viewer/2022050722/54990eb6b4795962238b45a5/html5/thumbnails/11.jpg)
Holton’s External Elements
![Page 12: Are we fulfilling our promise](https://reader034.fdocuments.net/reader034/viewer/2022050722/54990eb6b4795962238b45a5/html5/thumbnails/12.jpg)
Holton’s External Elements
![Page 13: Are we fulfilling our promise](https://reader034.fdocuments.net/reader034/viewer/2022050722/54990eb6b4795962238b45a5/html5/thumbnails/13.jpg)
Holton’s External Elements
![Page 14: Are we fulfilling our promise](https://reader034.fdocuments.net/reader034/viewer/2022050722/54990eb6b4795962238b45a5/html5/thumbnails/14.jpg)
ASSESSMENT METHODSHow exactly should we get this information?
![Page 15: Are we fulfilling our promise](https://reader034.fdocuments.net/reader034/viewer/2022050722/54990eb6b4795962238b45a5/html5/thumbnails/15.jpg)
The 5 Ws
![Page 16: Are we fulfilling our promise](https://reader034.fdocuments.net/reader034/viewer/2022050722/54990eb6b4795962238b45a5/html5/thumbnails/16.jpg)
Surveys / Feedback• All 4 levels
• Tips for sucessful surveys• Face-to-face• Concise• Organized / Consistent• Standardized vs. Customized• Most important questions first• Use open questions sparingly• Create realistic baseline expectations
![Page 17: Are we fulfilling our promise](https://reader034.fdocuments.net/reader034/viewer/2022050722/54990eb6b4795962238b45a5/html5/thumbnails/17.jpg)
Performance-based assessment
• Learning• Transfer designPerformance Step Yes No
Welcome and thank the audience
State purpose of the presentation
Give an outline of the presentation
Use discourse markers in outline
Mention admin rules (questions, breaks)
Signal the end of the introduction
Introduce the first topic using signposting
Maintain eye contact with audience
Speak clearly
Use appropriate body language
Total
Presentation Introduction
![Page 18: Are we fulfilling our promise](https://reader034.fdocuments.net/reader034/viewer/2022050722/54990eb6b4795962238b45a5/html5/thumbnails/18.jpg)
Client contact• Externalities• Behavior and results
• Targeted trainer-learner conversations• What’s going on in your department right now?• Are there any big projects coming up?• Did that lesson on trade fairs help you in Frankfurt? What helped
the most? How was it different than what we practiced?• Have you written any emails in English this week? Can you print it
for us?
• Client check-ins
![Page 19: Are we fulfilling our promise](https://reader034.fdocuments.net/reader034/viewer/2022050722/54990eb6b4795962238b45a5/html5/thumbnails/19.jpg)
BRINGING IT TOGETHERReporting
![Page 20: Are we fulfilling our promise](https://reader034.fdocuments.net/reader034/viewer/2022050722/54990eb6b4795962238b45a5/html5/thumbnails/20.jpg)
Ten inputs
1. Needs
2. Objectives
3. Content
4. Participants
5. Schedule
6. Facilities
7. Instructor(s)
8. Training materials
9. Program coordination
10. Program evaluation
![Page 21: Are we fulfilling our promise](https://reader034.fdocuments.net/reader034/viewer/2022050722/54990eb6b4795962238b45a5/html5/thumbnails/21.jpg)
4 levels
• Reaction – customer satisfaction
• Learning – improved knowledge and skills
• Behavior – application outside the classroom
• Results – impacts on the business
![Page 22: Are we fulfilling our promise](https://reader034.fdocuments.net/reader034/viewer/2022050722/54990eb6b4795962238b45a5/html5/thumbnails/22.jpg)
Externalities
• Motivation elements• Ability elements• Environment elements
![Page 23: Are we fulfilling our promise](https://reader034.fdocuments.net/reader034/viewer/2022050722/54990eb6b4795962238b45a5/html5/thumbnails/23.jpg)
Recommendations
• What is impacting our assessment?• 10 Inputs?• External factors?
• How do we improve?• Change an input?• Influence an external
factor?
• What should we share internally?
• What should we share externally?
![Page 24: Are we fulfilling our promise](https://reader034.fdocuments.net/reader034/viewer/2022050722/54990eb6b4795962238b45a5/html5/thumbnails/24.jpg)
INTO PRACTICEWorkshopping the four levels
![Page 25: Are we fulfilling our promise](https://reader034.fdocuments.net/reader034/viewer/2022050722/54990eb6b4795962238b45a5/html5/thumbnails/25.jpg)
Audience survey - Reaction
http://vot.rs/5ab67c
![Page 26: Are we fulfilling our promise](https://reader034.fdocuments.net/reader034/viewer/2022050722/54990eb6b4795962238b45a5/html5/thumbnails/26.jpg)
Focus groups - Learning
Discuss:• Three things you learned from today’s sessions
• 4 Levels• Don‘t assume transfer (reality checks)• Learner motivation questions
• One thing you would like to hear more about• Operationaizing it. (examples)
![Page 27: Are we fulfilling our promise](https://reader034.fdocuments.net/reader034/viewer/2022050722/54990eb6b4795962238b45a5/html5/thumbnails/27.jpg)
Focus groups - Behavior• Discuss how you will apply what you have learned from
today’s sessions.• Focus group results:
• Review the feedback form• Be more proactive about asking for achievements and successes• Look for testing service (before and after)• 3 months later, what has stuck? (before and after rating)• Use the worksheet as basis
![Page 28: Are we fulfilling our promise](https://reader034.fdocuments.net/reader034/viewer/2022050722/54990eb6b4795962238b45a5/html5/thumbnails/28.jpg)
Focus groups - Results• What impact on your organization do you expect to see as
a result of today’s training sessions?
![Page 29: Are we fulfilling our promise](https://reader034.fdocuments.net/reader034/viewer/2022050722/54990eb6b4795962238b45a5/html5/thumbnails/29.jpg)
TAKING IT FROM HEREConclusion
![Page 30: Are we fulfilling our promise](https://reader034.fdocuments.net/reader034/viewer/2022050722/54990eb6b4795962238b45a5/html5/thumbnails/30.jpg)
References• Holton III, E. F. (1996, Spring). The Flawed Four-Level Evaluation Model. Human
Resource Development Quarterly, 7(1), 5-21.• Kirkpatrick, D. L. (1994). Evaluating Training Programs: The Four Levels. San
Francisco: Berrett-Koehler.• Kirkpatrick, J. (2007, Aug). The Hidden Power of Kirkpatrick's Four Levels. T+D
Magazine, pp. 34-37.• Kirkpatrick, J. (2011, Spring). The New World Kirkpatrick Four Levels. The Canadian
Learning Journal, pp. 19-21.• Maister, D. H. (1993). Managing the Professional Services Firm. New York: Simon &
Schuster.• O'Toole, S. (2009, Aug). Kirkpatrick on Evaluation: Not Crazy After All These Years.
Training and Development in Australia, pp. 23-25.• Praslova, L. (2010). Adaptation of Kirkpatrick's Four Level Model of Training Criteria to
Assessment of Learning Outcomes and Program Evaluation in Higher Education. Educational Assessment, Evaluation & Accountability(22), 215-225.
• Steensma, H., & Groeneveld, K. (2010). Evaluating a Training Using the "Four Levels Model". Journal of Workplace Learning, 22(5), 319-331.
• Stoel, D. (2004, Jan). The Evaluation Heavy Weight Match. T+D Magazine, pp. 46-48.
![Page 31: Are we fulfilling our promise](https://reader034.fdocuments.net/reader034/viewer/2022050722/54990eb6b4795962238b45a5/html5/thumbnails/31.jpg)
For more information / questions
Contact: Charles Rei
http://www.reicommunicationtraining.com
@Charlesrei1
Presentation and handouts at Business English Ideas
http://businessenglishideas.blogspot.de/
Ongoing conversation on the BESIG Ning
http://iatefl-besig.ning.com/
![Page 32: Are we fulfilling our promise](https://reader034.fdocuments.net/reader034/viewer/2022050722/54990eb6b4795962238b45a5/html5/thumbnails/32.jpg)
Quality processes• If you were the client, would you care about these issues?• Do any of these questions represent an unreasonable
client expectation?• How confident do you feel that your organization would
outperform the competition on these questions?• How valuable would it be if your organization earned top
scores on this consistently?• Can you think of another way to ensure consistent
performance in these areas?
Thank you!