Researching the Effectiveness Researching the Effectiveness of Serious Games: Asking the of Serious Games: Asking the

Right QuestionsRight Questions

Dr. Jennifer SolbergDr. Jennifer Solberg

US Army Research Institute for theUS Army Research Institute for theBehavioral and Social SciencesBehavioral and Social Sciences

Serious games are being increasingly incorporated into learning environments.

Examples:• Classroom Education• Games for Health• Military Training

What sorts of questions should we be asking?Really, there’s only one.

The Rise of Serious GamesThe Rise of Serious Games

The One Big QuestionThe One Big Question

Do Serious Games Work? Are serious games an effective means of training?

Are they effective above and beyond current methods?

It’s that simple!

• If not, nothing else matters

It’s not that simple!

• Many factors are undoubtedly involved in the effectiveness of serious games.

OverviewOverview

Current thought on the One Big Question (OBQ)

• Answers so far

• Are we asking the right questions?

What questions we should be asking

• The Player

• The Game

• The Process

Where we can find the answers

• Performance within and outside the game

• Individual difference measures

• Preference measures

Where do we go from here?

Current Thought on the OBQCurrent Thought on the OBQ

Is there evidence that serious games are effective for training?

• Sometimes.

Is there evidence that serious games in general are effective for training above and beyond traditional methods of instruction ?

• Not really. (Hays, 2005)

What Does this Mean?

Current Thought on the OBQCurrent Thought on the OBQ

“The empirical research of the instructional effectiveness of games is fragmented, filled with ill defined terms, and plagued with methodological flaws.” (Hays, 2005)

“There appears to be a consensus among a large number of researchers with regard to the negative, mixed, or null findings of game research, suggesting that the cause might be a lack of sound instructional design embedded in the games” (O’Neil et al, 2005)

“Successful learning in serious games involves a complex interaction between the player and the game. In order to develop serious games that work, we must ask the right questions about this interaction.” (Solberg, this week)

Asking the Right QuestionsAsking the Right Questions

Any type of learning involves three components: the player, the instructional material, and the learning process itself.

We need to ask questions about all three.

The Player: What characteristics of the player make them more likely to learn through serious games?

The Game: What aspects of the game itself can we manipulate to maximize effectiveness?

The Process: In what context should the game be delivered to the player to maximize the likelihood of learning?

Is a game-based environment conducive to our normal learning and memory processes?

• Kirschener et al (2006). The trend of relying on unstructured learning is contrary to cognitive architecture.

• Clark’s talk yesterday

Are serious games conducive to normal interactions in a multi-player setting?

• Method of communication

• Restriction of actions

The PlayerThe Player

The PlayerThe Player

Do some people have characteristics that lend themselves to success in game-based learning? You may be surprised!

Examples:Sex

•Carr (2005). Female players avoid female avatars in game environments.

Age/Generation Cognitive ability Personality

•Learning style preferencePropensity to play games

•Orvis et al. (2005, 2006). Cadets’ gaming experience

Prior experience with training material

The GameThe Game

How can we design serious games to maximize the likelihood of learning?

Current thought:– Serious games should be tied to structured learning objectives.– Serious games should be motivational.

What aspects of serious games can we manipulate to assess effectiveness?

•Modality (of instructions, feedback) •Type of instructional material

•Changes in difficulty level •Complexity (part v. whole task)

•Transparency of instructional material •Duration (effects on motivation)

•In game feedback/score •Help (scaffolding, availability)

The ProcessThe Process

Does the way in which serious games are administered affect the quality of learning?

Possible factors:

• Pre-game instruction

(game play and training domain)

• Presence of instructor

• Feedback during/after game

(modality and timing)

• Effects of retesting

We have lots of questions. Where can we find the answers?

Finding the AnswersFinding the Answers

Where can we find the answers?

• Performance measures within and outside the game

• Individual difference measures

• Preference measures

These measures make up the types of data used in serious games research.

– It’s important to consider including measures in the game design so that research is possible later on.

• Validation

• Improvement

• Trouble-shooting

Performance MeasuresPerformance Measures

Within-game: How well do players play the game?• Score • Time to completion• Qualitative decisions/actions• Relative position to others?• Communication between players

Outside the game: How well do players transfer skills to other tasks?

• “Real world” taskNolan & Jones (2005) squad exercise performance after

training with COTS game• Tests

Retention, recognition– Training domain knowledge– KSAs

Individual Difference TestsIndividual Difference Tests

In order to determine the extent to which player characteristics can affect learning in serious games, measures of these characteristics should be considered.

• Perceptual/attentional tests

• Cognitive factors (e.g. decision-making ability, memory span)

• Personality factors (e.g. extraversion, conscientiousness, learning style preference)

Preference MeasuresPreference Measures

An important aspect of serious games is their ability to motivate. Gaming should be fun.

Preference measures provide data on the enjoyability of the game.

Also,

• Preference for game-based instruction

• Preference for modality of instruction (e.g. visual v. audio)

Too Much Information?Too Much Information?

How much performance information is enough? What kind is most informative?

This is a topic of debate (even in my lab), and largely depends on constraints of project

• Subjectivity v. objectivity

• Need for rapid deployment (especially in military)

• Need for troubleshooting

• Need to answer theoretical questions

Should there be a “standard” methodology?

Where Do We Go Now?Where Do We Go Now?

Assessing the answer to the OBQ will require asking questions about the interaction between

• The Player

• The Game

• The Process

Some answers can be found in:

• Performance measures within and outside the game

• Individual difference measures

• Preference measures

Where Do We Go Now?Where Do We Go Now?

These are all empirical questions that need to be answered!

Serious games are being implemented whether or not we answer the OBQ!

Non-trivial training domains

• Military training

• Medical/health instruction

An AfterthoughtAn Afterthought

The Author

• How easily does the author of the game interact with the authoring interface?

– May affect ability to incorporate training objectives into game environment.

• How much control does the author need?

– Sophistication of scenarios v. accessibility

– Decision based on finances, access to facilities, need for rapid deployment