Sharing Lessons on Learning Metrics
LINGOs Annual Meeting Sept. 23-24, 2009Steven Honeyman and Marie-Laure CurieGlobal Capacity Building Department
Let’s start with small group discussions• Briefly describe your learning metrics (overview only)• Discuss advantages and disadvantages of different
components • Summarize what the group thinks are some new or
interesting ideas the whole group should know about
LINGOs Annual Meeting - Sept. 23-24, 2009
Kirkpatrick’s model
Includes four levels/steps of outcome evaluation:• Level 1—Reaction – satisfaction survey• Level 2—Learning – tests • Level 3—Behavior – on-the job behavior change
survey• Level 4—Results – business outcomes
LINGOs Annual Meeting - Sept. 23-24, 2009
Measuring Training Impact at PSI
SMILE - Social Marketing Initiative for Learning and Excellence
Introduction to the PSI Behavior Change Framework
• Behavior change framework drives PSI marketing activities– Fundamental to social marketing
• Applicable to all behaviors promoted by PSI• Based on behavior change theory• Determinants measured through TRaC (Tracking
Results Continuously) survey
LINGOs Annual Meeting - Sept. 23-24, 2009
PRODUCT PRICE PLACE PROMOTION
SOCIAL MARKETING INTERVENTION
OPPORTUNITY ABILITY MOTIVATION
POPULATION CHARACTERISTICS
AT RISK POPULATIONS
RISK REDUCING BEHAVIOR, PRODUCT OR SERVICE USE
IMPROVED HEALTH STATUS/QUALITY OF LIFE
Availability
Brand Attributes
Locus of control
Willingness to pay
Outcome expectation
Belief
Threat
Attitudes
Intentions
Subjective norm
Self Efficacy
Social Support
Knowledge
Social Norm
Quality of Care
Brand Appeal
The overall framework
LINGOs Annual Meeting - Sept. 23-24, 2009
PRODUCT PRICE PLACE PROMOTION
SOCIAL MARKETING INTERVENTION
OPPORTUNITY ABILITY MOTIVATION
POPULATION CHARACTERISTICS
AT RISK POPULATIONS
RISK REDUCING BEHAVIOR, PRODUCT OR SERVICE USE
IMPROVED HEALTH STATUS/QUALITY OF LIFE
Availability
Brand Attributes
Locus of control
Willingness to pay
Outcome expectation
Belief
Threat
Attitudes
Intentions
Subjective norm
Self Efficacy
Social Support
Knowledge
Social Norm
Quality of Care
Brand Appeal
What PSI does…
Our sphere of influence
LINGOs Annual Meeting - Sept. 23-24, 2009
PRODUCT PRICE PLACE PROMOTION
SOCIAL MARKETING INTERVENTION
OPPORTUNITY ABILITY MOTIVATION
POPULATION CHARACTERISTICS
AT RISK POPULATIONS
RISK REDUCING BEHAVIOR, PRODUCT OR SERVICE USE
IMPROVED HEALTH STATUS/QUALITY OF LIFE
Availability
Brand Attributes
Locus of control
Willingness to pay
Outcome expectation
Belief
Threat
Attitudes
Intentions
Subjective norm
Self Efficacy
Social Support
Knowledge
Social Norm
Quality of Care
Brand Appeal
What we try to influence…
LINGOs Annual Meeting - Sept. 23-24, 2009
PRODUCT PRICE PLACE PROMOTION
SOCIAL MARKETING INTERVENTION
OPPORTUNITY ABILITY MOTIVATION
POPULATION CHARACTERISTICS
AT RISK POPULATIONS
RISK REDUCING BEHAVIOR, PRODUCT OR SERVICE USE
IMPROVED HEALTH STATUS/QUALITY OF LIFE
Availability
Brand Attributes
Locus of control
Willingness to pay
Outcome expectation
Belief
Threat
Attitudes
Intentions
Subjective norm
Self Efficacy
Social Support
Knowledge
Social Norm
Quality of Care
Brand Appeal
To change a behavior...
LINGOs Annual Meeting - Sept. 23-24, 2009
PRODUCT PRICE PLACE PROMOTION
SOCIAL MARKETING INTERVENTION
OPPORTUNITY ABILITY MOTIVATION
POPULATION CHARACTERISTICS
AT RISK POPULATIONS
RISK REDUCING BEHAVIOR, PRODUCT OR SERVICE USE
IMPROVED HEALTH STATUS/QUALITY OF LIFE
Availability
Brand Attributes
Locus of control
Willingness to pay
Outcome expectation
Belief
Threat
Attitudes
Intentions
Subjective norm
Self Efficacy
Social Support
Knowledge
Social Norm
Quality of Care
Brand Appeal
That improves health
LINGOs Annual Meeting - Sept. 23-24, 2009
OAM
• Opportunity: institutional or structural factors that influence an individual’s chance to perform a promoted behavior.
• Ability: an individual’s skills or proficiencies needed to perform a promoted behavior.
• Motivation: an individual’s desire to perform a promoted behavior.
LINGOs Annual Meeting - Sept. 23-24, 2009
• What is it?– Repetitive quantitative survey that measures behavior change
of a specific behavior of a specific target group– Measure the effectiveness of project interventions and
activities• Why do we use it?
– To better design programs, to monitor program progress (ensure that the target population is adopting the promoted behavior and if needed, make adjustments) and evaluate how we did (target group and behavior)
– Informs decision-maker where to invest more/fewer resources• Who does it?
– All PSI programs • When does it occur? - Prior, during (sometimes often) and at the end of an
intervention
TRaC (Tracking Results Continuously))
LINGOs Annual Meeting - Sept. 23-24, 2009
CAPACITY BUILDING INTERVENTION
OPPORTUNITY ABILITY MOTIVATION
POPULATION CHARACTERISTICS
BY PSI STAFF
USE OF SKILLS ACQUIRED AT THE TRAINING
IMPROVED JOB PERFORMANCE AND OVERALL IMPACT OF PSI PROGRAMS
Availability
Brand Attributes
Locus of control
Willingness to pay
Outcome expectation
Belief
Threat
Attitudes
Intentions
Subjective norm
Self Efficacy
Social Support
Knowledge
Social Norm
Quality of Care
Brand Appeal
The overall Capacity Building framework
LINGOs Annual Meeting - Sept. 23-24, 2009
• What is it?– Short, quantitative survey– Measures application of training on-the-job
• Why do we use it?– Programmatic check to ensure that the training had the
intended impact of transferring needed skills/behaviors to staff and to identify gaps in training content/delivery
– Informs CB decision-maker where to invest more/fewer resources
• Who does it?– Global Capacity Building department (for virtual training as
well)• When does it occur?
– Approx. 6 to 8 weeks after the training ended
TRaC-T (Tracking Results Continuously for Training))
LINGOs Annual Meeting - Sept. 23-24, 2009
Kirkpatrick’s model
Includes four levels/steps of outcome evaluation:• Level 1—Reaction – satisfaction survey• Level 2—Learning – tests • Level 3—Behavior – TRaC-T survey• Level 4—Results – business outcomes
LINGOs Annual Meeting - Sept. 23-24, 2009
TRaC-T survey (example)
LINGOs Annual Meeting - Sept. 23-24, 2009
LINGOs Annual Meeting - Sept. 23-24, 2009
Some TRaC-T results
TRaC-T Budget Review training - August 5 & 6 2009
LINGOs Annual Meeting - Sept. 23-24, 2009
LINGOs Annual Meeting - Sept. 23-24, 2009
Some TRaC-T results
TRaC-T might have some limitations:• Self reported behavior, • Needs to be built into the training program,• Needs resources to develop and manage, • May not apply learning right away.
But it has many potential benefits:• Demonstrates training effectiveness to senior managers, • Convinces supervisor of the value of a particular training, • Allows targeting of training resources to highly effective training,• Provides feedback to those leading training on whether they are
being effective, and• Can be automatically set-up for online training !
LINGOs Annual Meeting - Sept. 23-24, 2009
For more information:
• www.psi.org/research/• Marie-Laure Curie, Manager, GCB Dept.,
[email protected]• Steven Honeyman, Director, GCB Dept.
LINGOs Annual Meeting - Sept. 23-24, 2009
Top Related