Game-based statistical assessments: Bypassing the self-serving biases

Thomas E. Heinzen, William Paterson University

David Plested, William Paterson University Susan Nolan, Seton Hall University

NJEdge, November, 2016, Princeton, NJ

Step 1: Organize yourselves into five groups.

Step 2: You have just created what game designers call a “guild” that will increase commitment to and responsibility for each other.

Step 3: Congratulations. You have experienced how one principle of game design can increase student retention.

You are less likely to leave this workshop if you have had some say in how you are organized into a small group, compared to a large group.

Now for a brief tribute to Johnny Carson:

Heeeeere’s Susan!

Agreement of Mutually Assured Mediocrity

Mindsets and Biases

Fluency Illusion

False Consensus Effect

Stereotype Threat

Stereotype Threat

0

2

4

6

8

10

12

Stereotype threat No stereotype threat

Mean Items Solved (Adjusted for SAT)

Blacks Whites

Getting Around Biases

1) Active (not lecture!)

Making Learning Stick

“Active learning engages students in the process of learning through activities…as opposed to passively listening to an expert. It emphasizes higher-order thinking…”

1) Active (not lecture!)

2) The testing effect

Making Learning Stick

0%

20%

40%

60%

80%

100%

5 minutes 2 days 1 week

Test Score

Studied Practice test

1) Active (not lecture!)

2) The testing effect

3) Repetition

Making Learning Stick

1) Active (not lecture!)

2) The testing effect

3) Repetition

4) Feedback

Making Learning Stick

1) Active (not lecture!)

2) The testing effect

3) Repetition

4) Feedback

5) Errorful learning/desirable difficulty

Making Learning Stick

What can you infer from this information trail?

See Gillath et al. (2012) and Gosling et al. (2002)

The Game-Based Experience Lab: Applying principles of game design to increase student retention.

Five Game Design Principles

1. Focus on the experience we are delivering.

2. Take advantage of techniques for onboarding.

3. Create opportunities for leveling up.

4. Design obstacles that create a flow zone.

5. Make decisions within the framework of a story.

Games and Game Design

GAME

Intended User

Experience

Actual User

Experience

Experienced by

Designer User

Motivation and Engagement

Games in Non-Game Contexts

GBA

Games

Gamification Game-like Situations

Games vs GBAs

Gameful Design Gameful design focuses on the experience of

those interacting with non-game systems (Deterding, 2015).

Without gameful design, gamification is useless. Gameful design: intrinsically motivating Gamification: extrinsic rewards

Gameful design has empirical backing (Roepke et al., 2015).

Stealth Assessments

TAPAS (Metadata)

Assessment Trinity/Trifecta

Authentic Assessment

Validity

Reliability

Fairness

Post-Validity Assessments

The Lens of Endogenous Value

Ludus and Paidia

ACTIVITY

Now, make college the game.

Guidance:

1. Use as many or as few materials as you like.

2. Have fun.

Experiencing Road Signs: Is this first impression

a metaphor for student retention?

A statistics course used the principle of leveling up to create a flow zone.

Week 1: Enter data in appropriate columns (10 points)

Week 2. Enter data in appropriate columns (10 point) Compute frequencies and display histograms (15 points)

Week 3. Enter data in appropriate columns (10 points) Compute frequencies and display histograms (15 points) Display descriptive statistics (20 points)

Post course comments (I should have been offended):

1. Trial and error were good. There was no real help along the way, but there were no consequences. (Failing forward)

2. I like that we got to leave early if we learned all we needed to know for that week. (Levels)

3. I liked having someone other than the professor teach me; it was much better. (Self-efficacy)

Post course comments (I should have been offended):

Q: When, if ever, did you experience something like an “epic win”?

4. When I began to be able to do the material and solve problems alone. (Intrinsic motivation)

5. Whenever I finished a new level I felt that I accomplished it. (Flow zone)

6. Completing the practice tests and doing them correctly. (Mastery learning) As professor, I had become unnecessary for their learning!

Student I.D Card Stickers! (Rewards for Leveling Up)

Leveling Up

Automatic Feedback loops

Discreet rewards

Implicit badges

Rewarding achievement

Self-Monitoring achievement

2.76

1.97

0.27

Game-based design University Design DegreeWorks

Figure 2. Student Preferences for Degree Audit Software

(screenshots only)

A Content Analysis of Fat Points and Fairness (probably related to failing forward).

0

25

53.8

63.6

16.7

0 0

10

20

30

40

50

60

70

Possible Points: 1 Possible Points: 100 Possible Points:100,000

Effect of Total Possible Points on Student Perceptions of a Course

(% students describing course as...)

Fair

Unfair

Help Donny (A Knowledge of Psychology Story)

Designed by Sam Richman (User Experience Strategist)

College student “Donny” is experiencing overwhelming stress and has decided to go into therapy.

Use your knowledge of psychology, even if it contradicts your intuition, to help Donny make some decisions.

Donny is taking a test but he is unsure of his answers and keeps second guessing himself.

What course of action should Donny take? A) Stick with his first hunch. B) Go over the questions he is worried about and change them if he thinks another answer might be right.

What is the single most joyful moment experienced in the life of a college student?

Graduation Joy.

Graduation joy is a shared experience.

Joy is a motivating

experience.

Graduation joy has generational consequences.

Shared graduation joy helps create loyal alumni.

Why do we deny students the experience of achievement joy until it is too late to help them?

A photo study of graduation joy delivers timely experiences of achievement joy

based on leveling up.

ACTIVITY Each group choose one game design principle to craft an

intervention to increase student completion.

Five Game Design Principles

1. Always focus on the experience we are delivering.

2. Take advantage of techniques for onboarding.

3. Create opportunities for leveling up.

4. Design obstacles that create a flow zone.

5. Make decisions within the framework of a story.