OLF1206 101 Thalheimer

Beyond Smile Sheets: Learning Audits and

Evaluations We Can Trust Will Thalheimer, Work-Learning Research

Produced by

December 7 & 8, 2006

101

Designing & Developing Online Assessments & Evaluations

Designing & Developing Online Assessments and Evaluations


Page 1Session 101 – Beyond Smile Sheets: Learning Audits and Evaluations We Can Trust, Will Thalheimer, Work-Learning Research

1

© Copyright 2006 Work-Learning Research, Inc.

Beyond Smile Sheets:

Will Thalheimer, PhDWork-Learning Research, Inc.

www.work-learning.comwww.willatworklearning.com

www.learningaudit.com

eLearning GuildDecember 2006

Some Research-Based Considerations

2


Work-Learning Research, Inc.

MissionHelp Clients Build More Effective Learning Bridge Gap between Research & PracticeCompile Research, Share Findings

WorkLearning AuditsWorkshopsResearch




Smile Sheet Example

4


Four Levels (from Donald Kirkpatrick)

1. Reactionsmilesheets

2. Learningretrieval,

decision-making

3. Behavior on-the-jobperformance

4. Results businessresults




5


Percent of Companies Using

0 20 40 60 80 100

Reaction

Learning

Behavior

Results

1.

2.

3.

4.

79.5 %

35.0 %

15.3 %

8.8 %

ASTD’s industry reports

6


For your most recent learning assessment, where did you give the learners the assessment?

A. In the training room/context.

B. In a different room/context.

C. Never done / Can’t remember.




7


For your most recent learning assessment, when did you give the learners the assessment?

A. At the end of training.

B. After a delay.

C. At end—and after a delay.

D. Never done / Can’t remember.

What do you What do you think aboutthink about

SMILE SHEETS?SMILE SHEETS?Type Your Answer in Chat

What do you think about Smile Sheets?




9


If smile-sheet ratings are high, what does that mean?

A. Learning results will likely be very good.

B. Learning results will likely be at least reasonably good.

C. Learning results cannot be predicted by smile sheets.

10


Correlation between levels?

Level 2to

Level 3

r=.12

Level 1to

Level 3

r=.16

Level 1to

Level 2

r=.09

Alliger, Tannenbaum, Bennett, Traver, & Shotland (1997). A meta-analysis of the relations among training criteria.

Personnel Psychology, 50, 341-357.

Very Weak Relationship between Levels




11


“Did You Like Training” vs.

“Is the Training Valuable?”

Level 1to

Level 3

r=.07

Level 1to

Level 2

r=.02

Level 1to

Level 2

r=.15

Level 1to

Level 3

r=.20

Asking how much learners like training

Asking how valuabletraining was

BeyondBeyondSmile SheetsSmile Sheets

Beyond Smile Sheets




13


How We Learn

fosse

ditchRehearsal

“A fosse is a ditch.”

Long-Term Memory

14


How We Retrieve Information

fosse

ditchSearch

Cue

Rehearsal

“A fosse is a ditch.”

Long-Term Memory




15


Learning Versus Retrieval

Search

Cuefosse

ditchRehearsal

Action

Learning

Retrieval

fosse

ditchRehearsal

Action

16


Learner Fulfillment

Business Results

Learning to Performance/Results

Learner Retrieves

Learner Applies

Learner Learns

Learning Performance Results




17


Learner Fulfillment

Business Results

Transfer Breakdown

Learner Retrieves

Learner Applies

Learner Learns

Learning Performance Results

18


To be a good predictor of on-the-job retrieval, where should a learning evaluation be given?

A. In the training room/context.

B. In a different room/context.

C. It depends...




19


Tested in the learning room or in a different room. Does this matter?

Smith, S. M., Glenberg, A., & Bjork, R. A. (1978). Environmental context and human memory. Memory & Cognition, 6, 342-353.

0

5

10

15

20

25

Tested ina

DifferentRoom

Tested inthe

LearningRoom

20


To be a good predictor of on-the-job retrieval, when should a learning evaluation be given?

A. At the end of training.

B. After a delay.

C. At end—and after a delay.

D. It depends...




21


Time Sequence of How Learning Becomes Performance

2 weeks

Retention

On-the-jobPerformance

Performance

1st Event

2nd Event

3rd Event

Learning

22


Retrieval Curves

0

20

40

60

80

100

1 2 3 4 5 6 7 8 9 1 0 1 1 1 2 1 3 1 4 1 5

Retention Performance

Posttest

RememberingPretest

Learning




23


Without Workplace Learning

1 2 3 4 5 6 7 8 9 1 0 1 1

LearningForgetting

24


With Workplace Learning

1 2 3 4 5 6 7 8 9 1 0 1 1

Learning

WorkplaceLearning




25


With Workplace Learning

1 2 3 4 5 6 7 8 9 1 0 1 1

Learning

WorkplaceLearning

Forgetting

26


Which group will do better on the posttests?

Learner LearnsPretest Posttest Delayed

Posttest

Learner Learns

Posttest DelayedPosttest

Group 1

Group 2

Assume:(1) pretest covers same material as posttests;

(2) learners get NO feedback on pretest.




27


Research Example

40

50

60

70

Pressley, M., Tanenbaum, R., McDaniel, M. A., & Wood, E. (1990). What happens when University students try to answer prequestions that accompany textbook material?Contemporary Educational Psychology, 15, 27-35.

No Pre-Questions

Pre-Questions

Read

Pre-QuestionsAnswered

Wrong

Pre-QuestionsAnsweredCorrectly

Pre-Questions Answered

28


Tests should be authentic

Level A – Real-world

Level B – High-fidelity simulation

Level C – Scenarios

Level D – Memorization

Level E – Attendance

Level F – Affiliation

Shrock & Coscarelli, Criterion Referenced Test Development

Quasi-Certification

Certification

“Level C represents the last level of certification that can be considered to assess an ability to perform on the job.Level D represents the first quantum jump away from

fidelity in assessment and should be used with caution.”




29


Scenario-based Question

You’re a trainer at a large multinational firm and are

designing a course to teach new-hire high school

graduates word processing skills. You understand the

value of practice and want to create meaningful

practice exercises for your class. For example, for your

beginner class you want them to practice how to set

the margins. How should you design your practice

examples to create the best on-the-job remembering

of the concepts you will teach?

What do you think about this question?




31


What do you think about this question?

32


Legal Issues

1. Tests must validly predict job performance, and not create unfair disadvantages.

2. Reliable.

3. You may have to prove your test is valid.

A. Documentation.

B. Post-hoc validation is rarely upheld in court.

C. Starts with job analysis




33


How would you design authentic tests?

1. Test for person who learned MS Excel.

2. Test for person taking leadership class.

3. Test for person learning hair styling.

4. Test for person learning new-product knowledge in order to sell the product.

5. Test for restaurant allergy safety for wait staff.

6. Test for onboarding training.

34


Tests can produce learning benefits!

0

1

2

3

4

Nungester, R. J., & Duschastel, P. C. (1982). Testing versus review: Effects on retention. Journal of Educational Psychology, 74, 18-22.

Learn-Learn

Learn-Test




35


Do our assessments validly measure both learning and forgetting?

Completion.

Smile sheets.

Measures of learning tend toward the measurable, not the authentic.

Immediately at end of learning.

In the same room/context as learning took place.

36


Most Assessments We Do Cannot Provide Valid Feedback on Our Effectiveness!!

0 20 40 60 80 100

Reaction

Learning

Behavior

Results

ASTD’s industry reports

1.

2.

3.

4.

79.5 %

35.0 %

15.3 %

8.8 %




37


What doesWhat doesthis mean?this mean?

What does this mean?

Instructional Design & Development Process

Feedback for future efforts1 2 3 4

Feedback for current project1 2

Research-Based Instructional Design

Experience-Based Instructional Design Expert-Based Instructional Design

Common-Sense-Based Instructional Design

The Instructional Design & Development Process

Design&

Develop

Deploy&

Evaluate

Prepare,Analyze,

Set Goals

Knowledge,Wisdom,Biases





39


Questions

What questions do you have now?

Questions later:

Will ThalheimerWork-Learning Research, Inc.Somerville, Massachusetts, USAinfo@work-learning.com617-718-0067www.work-learning.comwww.LearningAudit.com

OLF1206 101 Thalheimer

Documents

Transcript of OLF1206 101 Thalheimer