Measuring Learning Effectiveness: Beyond Training Happy Sheets & Memory Recall
35
1 Thurs, April 14th, 2016 12 -1 pm AEST Ways to participate: • Q&A Box - comment, whinge & opinions • Twitter Backchannel - #lrncafe Knowledge Sharing Better Practices Experienced Panel
-
Upload
learningcafe-capabilitycafe -
Category
Business
-
view
197 -
download
0
Transcript of Measuring Learning Effectiveness: Beyond Training Happy Sheets & Memory Recall
1
Thurs, April 14th, 2016 12 -1 pm AEST Ways to participate:• Q&A Box - comment, whinge & opinions• Twitter Backchannel - #lrncafe
Knowledge Sharing
Better Practices
ExperiencedPanel
Introductions AbbVieABC AGLAMPANZ BankARCAAustralian Institute of managementAustralian Red CrossBank AustraliaBankwestBlackboardBNZBT Financial GroupCancer Council QueenslandCBACBH GroupCoca-Cola AmatilConcordia UniversityCUB PTY LTDDegreedDownerEnergy AustraliaEricssonErnst & YoungFACSFedEx ExpressFord-Werke GmbHHCF IAGIMC
IP AustraliaJustine JardineLynda Moe TransformationsManipal Global Ed ServMcGrathNiccolMelbourne Business SchoolNABNATANovartisPresence of ITPrimary Health CareQantasQantas Credit UnionQatar AirwaysQBEQueensland Public ServicesRACPRecoveriescorpRio TintoShine LaywersSiemens LtdSimplotSkillsoftSMS Management & TechSt John Ambulance WASuncorp TelstraToyotaTP3University of SydneyWestpacWVI
Attendees From140+
Vanessa Blewitt Learning Effectiveness Lead, Corporate Training and Learning – Nestle
Will Thalheimer President – Work-Learning Research, Inc
Jeevan Joshi Producer – LearningCafe
BlogMagazine
Webinar Discussions
UnConference
TwitterLinkedin
Coffee Catch Ups
Capability Building
Workshops
Community of Learning Professionalswith a focus on implementing ideas
BuildingL&D
Capability
Webinar Next Steps
Upcoming Webinars
Recorded Webinars
UnConference
Please subscribe
Next Webinar - VR in Learning – Has its time come ?http://bit.ly/lcwebmay16
• State of VR (Consumer & business)
• Use cases for business• How to develop a VR Learning
solution• Opportunities and challenges
Thurs May 12th, 2016 12 to 1 PM
Kain TietzelFounding CEO –Start
Plus others
Why you should attend• 2016 is supposed to be year of VR• L&D likely to asked about it.• VR Learning is being built.• Stay ahead of curve.
Melbourne UnConference – Aug 18, 2016Limited to 100. Tickets available. Sydney was sold outhttp://www.learningcafe.com.au/events
Tony GleesonCEO - AIRAH
Julie CatanachExec Manager Performance Suncorp Group
Peter HallHead L&DQBE - APAC
David Le PageComm Mgr - Melbourne
Andrew WrightWorldwide Intranet Challenge
Ideasatwork.com.au Collaborative explorations by Australian organisations facilitated by LearningCafe
Starting Soon
TinCan/xAPI
MOOCs
Content Curation for
Learning
? (Too soon)
Yes
Monthly Observations in Australian L&D*
•Significant restructuring in HR/ L&D.• Head count reductions• Getting new skills in the team.
•Gap in Business Acumen in L&D, a real worry.
•Major LMS & HRMS not keeping pace with flexibility and UX. • Number of point cloud solutions coming into the market .
•More focus on hard skills / business Learning.• Less on soft skills.
* Opinions based on discussions and interviews with senior L&D professionals. Not survey involved.
Webinar Dashboard
Start Up/Experiment
Organisation / IndustryAdoption
Why Bother ? Remove Hype
Maturity
Bus
ines
s Va
lue
Low
Low
High
High
Now
Novice Explorer Practitioner Expert
Get Answers
MakeSense
Looking for Answers
Expertise Level
Next Steps ?
Not Much A Lot
Vision
LearningCafe Expertise
Run workshops for L&D teams for• Leading Australian Bank• Leading Insurance CompanyConsulting on architecture for• Professional services firmCovered in UnConference 2015 + 2016
Questions from you
• What are 3 tips in gathering valuable data/statistics? • How do you measure accurately if for e.g. only 100 out of 500 people respond to a survey? When should you make assumptions from the data?
• Like quality, learning effectiveness is also a perception in the minds of key stakeholders (and influenced by their expectations). How do we best accommodate this reality in our measurement strategy?“
• Value in blended learning design - at the end, throughout, at the start or combination of?
HELICOPTER VIEW
Jeevan JoshiProducer – LearningCafe &Digital Learning Adviser – Fairfax Media
PRIMER
Analysis
Design
Development
Implementation
Evaluation
ADDIE
OK
Complex
Requires more time & effort
Difficult
Requires time & effort
* http://www.gosignmeup.com/kirkpatrick-phillips-model/
Kirkpatrick Model of Evaluation*
WE HAVE A PROBLEM
PROBLEMS
Too difficult to isolate impact of
Learning
Evaluation is not standardized
enough
Not enough budget and skills
Business moves fast and loses
interest.
MORE PROBLEMS
Rise of Informal Learning
Speed of Business in Increasing
L&D Budget Likely to
Decrease
Employee engagement is decreasing
TinCan/[email protected]
17
Intranet – Videos/podcastsDiscussion Boards (COP)(yammer)Wiki (Aussie /prod info/Wiki/IT)On job activities (Perf Management)DiscussionsTeam meetings (learning component)Other business systems
Online Courses on LMS + WebinarClassroom Courses/ Online CoachingOn Job Activities (competence demo)Assessments/surveys (e.g. survey monkey)SimulationPerformance Support ToolSecondment
Social MediaYouTubeFacebookWebsite – HBRAny web resourceExternal COPCoffee conversations
MOOCsExternal coursesFormal Education (Degrees)Formal Collaborative Program with UnisMemberships/prof bodiesFormal mentoring programsCertifications/exams
Formality
Sou
rces
InternalInformal
External
Formal
xAPI/ TinCan – Solution ?
Layer 1• Modern version of
SCORM
Layer 2• Record any
learning experience
Layer 3• Frees data from
LMS
Layer 4• Correlate job
performance and training data
http://tincanapi.com/the-layers-of-tin-can/
Happening Aspirational
Technical Strategic
Low readiness Maturity /capability
www.tincanatwork.com
ORGANISATIONAL REALITIES
Vanessa Blewitt - Learning Effectiveness Lead, Corporate Training and LearningNestle
Deciding to Measure Learning Effectiveness
What is the value of T&L investments?
Executive Board
• That was the easy part!
Starting Point: T&L as a “HR Thing”, an Event
• < 1 in 10 learners discuss expected outcomes with line manager.
• Alignment to business not clear.
• Alignment to learner development plan low.
• Key measure = • No on the job follow-up or support.
• Action plans un-actioned.
• No measures of what happened…or why.
• “Black box”.
Before During After
Look Familiar?
Defining “Value”
Guiding Principle:In any transaction, the supplier determines the price, but the purchaser determines the value.
In order to measure value, first need to create conditions for value.
Tools, Process, Measures
Design Delivery
What is the value of T&L investments?
Collect Nestle
Insights
Conduct Industry
Research
Develop Learning
Effectiveness Framework
Defining Learning Effectiveness Measures: Design
DesignWith the business for the business
Business Level Measures WHAT: What is expected?
What does success look like? ? How will you recognize it?? Why this learning…target which business
objectives & priorities?? What will you reflect on in 6 months time
to decide if learning was worthwhile/ added value?
? Expected outcomes? … observable / measurable? KPI impacts? Trends? Feedback? What should CHANGE?
WORKSHOPSenior Stakeholders
& AlumniHOW & WHO
INTAKE FORMSenior Stakeholders
& Alumni
INTERVIEWReflections & Feedback
Alumni @+200 days
Defining Learning Effectiveness Measures: Delivery
DeliveryFor on the job performance change
Learner Level MeasuresWHAT: What is expected? What happened?WHY: Why did it happen?
? Learning Alignment • To development plan & objectives
? Learning Impact on business• Expected outcomes e.g. KPIs
? Learner Development• On the job behaviors – do & say
? Learning Application • Action Plan status• Opportunities & Obstacles
? Learner Experience• Confidence• Commitment• Net Promoter Score
HOW & WHO
SurveysLearner with Line Manager
Option: HRBP Facilitation@ Before @After
SurveysLearner
@Nomination @During
No survey is called “survey”Learner centric questions
Importance of Why MeasuresExample of Actionable Insights
WHATHappened?
• Proficiency Gap Closure = 50% v Expectation (Target) of 80%.
WHYDid it happen?
• Action Plan Status = 60% Fully or Mostly Implemented.• Obstacles to apply learning on the job = 70% learners reported
higher/competing priorities.
WHATAction to take?
Current Cohort: Targeted business escalation, coaching. Future: Increase business to business communication re importance of
application to achieve business and career success, learn & leverage from strength business areas, add how to address higher/competing priorities to learning experience.
TargetSet benchmarks & change
variances
Track TrendsOver time.
Avoid “knee-jerk” reactions
TellTailor to stakeholders
Apply “So What?!” Filter
Challenges: Organisation &Process, Data, Enablers
Organisation • Identifying right stakeholders & getting them to the table.• Defining what success looks like in a way that is tangible &
linked to business.• Defining objectives to support learning effectiveness
measures (finite, verb, scalable for proficiency levels).
Process, Data, Enablers• Data access & quality for analytics.• Sophistication of requirements v enabler capabilities.• Managing surveys & reporting > timeliness, quality,
defining use cases.• Scalability.
Challenges: Climate & CultureClimate & Culture• Survey response rates (L&D to business).• Training as reward & recognition.• Fear of “why” measures (e.g. manager support
rate).• Learners: Lack accountability for self-development.
“Develop me.” “I was told to come”. • Line Managers: T&L = . Develop people not
include in bonus. “ I don’t have time”. • HRBPs: Operational focus. “Not My Job”, “Too
hard”, “Good training gets applied”, “Why measure?”.
• L&D: Event focused. Self-contained. Service oriented. Uncomfortable to challenge business. “Not what I’m asked for”, “No influence beyond event”, “What am I supposed to do with this?” “What if I’m told to “fix it?””
• ALL: OVERWORKED. OVERWHELMED. CHANGE RESISTANT.
Top 10 Key Learnings WHAT
“as expected?”
!As defined by stakeholders!
WHY“why / why
not?” Act to
achieve / exceed “as expected”
!As defined by stakeholders!
Effective learning effectiveness
measures
1. “What does success look like?” is a hard question to answer! ASK! Don’t ass-u-me.
2. “Fear” of what measures will reveal & accountability for action
3. Required culture change in L&D as much as business
4. Use inspiring facts & stories. Shine a light: “look familiar?”
5. Appetite exists & can be built. Don’t push. Leverage multiple channels to “build the army” of collaborators & early adopters.
6. Engaging stakeholders increases stakeholder engagement & accountability
7. Respectful challenge is well-received, build & helps earn a seat at the table
8. Alignment & behaviour change more valued than ROI
9. Focus on HOW more than WHAT
10. Long process of CHANGE. Be the tortoise.
+1 Start small & focussed
STRONG FOUNDATIONS
Will ThalheimerPresident – Work-Learning Research, Inc
Questions from you
• What are 3 tips in gathering valuable data/statistics? • How do you measure accurately if for e.g. only 100 out of 500 people respond to a survey? When should you make assumptions from the data?
• Like quality, learning effectiveness is also a perception in the minds of key stakeholders (and influenced by their expectations). How do we best accommodate this reality in our measurement strategy?“
• Value in blended learning design - at the end, throughout, at the start or combination of?
www.learningcafe.com.au
lrncafe
http://bit.ly/lcafefb
blogs
learning conversations
free resources
workshops
UnConference 2016Sydney Melbourne
Webinar recording, ebooks, L&D frameworks
Building Effective Employee Social Networks
33
Ideas@work Collaborations
Next Steps
Join Special Interest
Community
Attend WorkshopContent Curation & Dissemination
for Learning
Attend UnConference
MelbourneBrisbane
LearningCafeLinkedIn Subgroup
Register interest
www.learningcafe.com.au
Register interest
www.learningcafe.com.au
Or send us an email - [email protected]
Ways to participate
Q&A Box - comments, gripes & opinion
Twitter Backchannel - #lrncafe
Webinar Feedback – http://bit.ly/lcrate1
SessionRecorded
Next Steps
Join Special Interest
Community
Attend Workshops
Attend UnConference
MelbourneBrisbane
LearningCafeLinkedIn Subgroup
Register interest
www.learningcafe.com.au
Register interest
www.learningcafe.com.au
Or send us an email - [email protected]
