Engineering Class Home Pages - A Formative Evaluation of ... · Web view2005/11/13 · Webster et...

Navigation Maps and Problem Solving: revised 11/13/05

THE EFFECT OF NAVIGATIOM MAPS ON PROBLEM SOLVING TASKS

INSTANTIATED IN A COMPUTER-BASED VIDEO GAME

by

Richard Wainess

14009 Barner Ave., Sylmar, CA 91342PHONE/FAX 818-364-9419

A Dissertation Presented to theFACULTY OF THE GRADUATE SCHOOL

UNIVERSITY OF SOUTHERN CALIFORNIAIn Partial Fulfillment of the

Requirements for the DegreeDOCTOR OF PHILOSOPHY

(EDUCATION)

December 2005

Copyright 2005 Richard Wainess


ACKNOWLEDGEMENTS

I am eternally grateful to a number of people without whose generous

guidance, support, and encouragement I might not have realized the enormous

accomplishment of completing my dissertation and, ultimately, earning the

degree of Doctor of Philosophy in Education. Most directly related to these

accomplishments are the faculty. I would like to thank Dr. Richard Clark and Dr.

Janice Schafrik, for their roles as teachers and members of the committee of five.

I would like to thank Dr. Yanis Yortsos for his support as my outside committee

member. My thanks goes to Dr. Ed. Kazlauskas, not only for his roles as teacher

and committee of three member, but more importantly, as my masters’ advisor

and for his support throughout my graduate career. And my final thanks to faculty

goes to my advisor Dr. Harold O’Neil for exemplifying the roles of master and

mentor, for opening his arms and his heart to me. Harry was my teacher, my

advisor, and my guide, and I will value our friendship as it continues to grow.

Special thanks go to my colleagues, Dr. Claire Chen and Dr. Danny Shen

for their friendship, assistance, and support. I am also grateful to my son and

granddaughter for their pride in me, giving me my most cherished motivation for

being a role model. I also owe an immeasurable debt of gratitude to my parents

who have encouraged every endeavor I have undertaken, every goal I aspired to,

and every dream I followed, regardless of their true feelings. They allowed me to

reach far beyond what might have been reasonable, a freedom that has led to

now. And my final thanks and endless love go to my wife, Janet, for being there

ii


when I needed her, giving me more than I deserved, and making my dream our

dream.

Table of Contents

ACKNOWLEDGEMENTS ii

List of Tables vii

List of Figures x

Abstract xi

CHAPTER I: INTRODUCTION 1Background of the Problem 1Statement of the Problem 3Purpose of the Study 4Significance of the Study 5Research Questions and Hypotheses 7Overview of the Methodology 8Organization of the Report 9

CHAPTER II: LITERATURE REVIEW 10Cognitive Load Theory 11

Types of Cognitive Load 12Working Memory 16Long Term Memory 17

Schema Development 17Automation 18

Mental Models 19Elaboration and Reflection 19Metacognition 20Meaningful Learning 21Mental Effort 21

Mental Effort and Motivation 22Goals and Mental Effort 23Goal Setting Theory 24Goal Orientation Theory 25

Self-Efficacy 25Self-Efficacy Theory 26Expectancy-Value Theory 26Task Value 27

Problem Solving 27O’Neil Problem Solving Model 28

iii


Learner Control 30Summary of Cognitive Load 32

Games and Simulations 38Games 39Simulations 40Simulation-Games 41Games, Simulations, and Simulation-Games 41Video Games 44

Motivational Aspects of Games 45Fantasy 47Control and Manipulation 48Challenge and Complexity 49Curiosity 50Competition 51Feedback 52Fun 52

Play 53Flow 53Engagement 54

Learning and Other Outcomes from Games and Simulations 54Positive Outcomes from Games and Simulations 55Relationship of Motivation to Negative or Null Outcomes

from Games and Simulations 63Relationship of Instruction Design to Learning

from Games and Simulations 64Reflection and Debriefing 65

Summary of Games and Simulations 65Assessment of Problem Solving 70

Measurement of Content Understanding 70Measurement of Problem Solving Strategies 75Measurement of Self-Regulation 76Summary of Problem Solving Assessment 77

Scaffolding 79Graphical Scaffolding 81Navigation Maps 82Contiguity Effect 89Split Attention Effect 89Summary of Scaffolding 90

Summary of the Literature Review 94

CHAPTER III: METHODOLOGY 103Research Questions and Hypotheses 103Research Design 104Study Sample 105

Pilot Study Sample 105

iv


Main Study Sample 105Solicitation of Participants for the Main Study 105Randomized Assignment for the Main Study 107Number of Participants Whose Data Were Analyzed 108

Hardware 110Instruments 112Demographic, Gameplay, and Game Preference Questionnaire 112Task Completion Form 114Self-Regulation Questionnaire 116SafeCracker® 116Navigation Map 119Knowledge Map 123

Content Understanding Measure 124Scoring of Knowledge Map 128

Domain-Specific Problem Solving Strategies Measure 130Scoring of Problem Solving Strategies Retention and Transfer Responses 136Procedure for the Pilot Study 138Administration of Demographic and Self-Regulation Questionnaires 138Introduction to Using the Knowledge Mapping Software 139Introduction to the Game SafeCracker 141

SafeCracker Training Script 142Introduction to Using the Navigation Map 146

Training Map Script 147Script for the Control Group on How to Navigate the Mansion 150First Game 151Creating the Knowledge Map (Occasion 1) 152First Problem Solving Strategies Questionnaire (Occasion 1) 152Second Game 153Knowledge Map and Problem Solving Strategies Questionnaires (Occasion 2) 154Debriefing and Extra Play Time 154Timing Chart for the Pilot Study 155Results of the Pilot Study 155

Adjustments to the Knowledge Mapping Instructions 156Adjustments to the SafeCracker Instructions 157Adjustments to the Problem Solving Strategies Instructions 159Adjustments to the Task Completion Form 160

Procedure for the Main Study 161Demographic and Self-Regulation Questionnaires 161Introduction to the Using Knowledge Mapping Software 161Introduction to the Game SafeCracker 165

SafeCracker Training Script 166Introduction to Using the Navigation Map 173

Training Map Script 174Script for the Control Group on How to Navigate the Mansion 177First Game 178

v


Creating the Knowledge Map (Occasion 1) 179Problem Solving Strategies Questionnaire (Occasion 1) 179Second Game 180Knowledge Map and Problem Solving Strategies Questionnaires (Occasion 2) 181Debriefing and Extra Play Time 182Timing Chart for the Main Study 182

CHAPTER IV: ANALYSIS AND RESULTS 184Research Hypotheses 184Content Understanding Measurement 185Interrater Reliability of the Problem Solving Strategy Measure 188Problem Solving Strategy Measure 193

Retention Question 193Transfer Question 198

Trait Self-Regulation Measure 200Safe Cracking Performance 205Continuing Motivation Measure 210Tests of the Research Hypotheses 211

CHAPTER V: SUMMARY OF RESULTS AND DISCUSSION 215Summary of Results 215Discussion 218Possible Effects from the Contiguity Effect and Extraneous Load 218Possible Effects from Strategy Training 223

Strategy Priming During Knowledge Map Training 225Strategy Priming During SafeCracker Training 225Strategy Priming During Navigation Map and BasicNavigation Training 226Strategy Priming at the Start of Each Game 226

Summary of the Discussion 227

CHAPTER VI: SUMMARY, CONCLUSIONS, AND IMPLICATIONS 230Summary 230Conclusions 235Implications 236

REFERENCES 240

APPENDIX A: Self-Regulation Questionnaire 257APPENDIX B: Knowledge Map Specifications 260

vi


List of Tables

Table Page

1. Characteristics of Games and Simulations 44

2. Non-Empirical Studies: Media, Measures, and Participants 57

3. Empirical Studies: Media, Measures, and Participants 60

4. Characteristics of Games, Simulations, and SafeCracker 118

5. An Example of Participant Knowledge Map Scoring 125

6. Problem Solving Strategy Retention and Transfer Questions 132

7. Idea Units for the Problem Solving Strategy Retention Question 134

8. Idea Units for the Problem Solving Strategy Transfer Question 136

9. Time Chart for the Pilot Study 146

10. Time Chart for the Main Study 183

11. Descriptive Statistics of Knowledge Map Occasion 1 and Occasion 2Scores for the Control Group, Navigation Map Group, andBoth Groups Combined 185

12. Descriptive Statistics of the Percentage of Knowledge Map Occasion 1and Occasion 2 Scores for the Control Group, Navigation Map Group,and Both Groups Combined 187

13. Knowledge Map Means by Group by Occasion 188

14. Matrix of the Number of Participant Responses Assigned to EachIdea Unit in the Problem Solving Retention Measure Based onTwo Rater’s Scoring 190

15. Matrix of the Number of Participant Responses Assigned to EachIdea Unit in the Problem Solving Transfer Measure Based onTwo Rater’s Scoring 192

16. Descriptive Statistics of Problem Solving Strategy Retention Occasion 1and Occasion 2 Scores for the Control Group, Navigation Map Group,and Both Groups Combined 194

vii


17. Descriptive Statistics of the Percentage of Problem Solving StrategyRetention Occasion 1 and Occasion 2 Scores for the Control Group,Navigation Map Group, and Both Groups Combined 196

18. Means for Problem Solving Strategy Retention by Group by Occasion 197

19. Descriptive Statistics of Problem Solving Strategy Transfer Occasion 1and Occasion 2 Scores for the Control and Navigation Map Groups 198

20. Descriptive Statistics of the Percentage of Problem Solving Strategy Transfer Occasion 1 and Occasion 2 Scores for the Control Group, Navigation Map Group, and Both Groups Combined 199

21. Means for Problem Solving Strategy Transfer by Group by Occasion 200

22. Descriptive Statistics of Trait Self-Regulation Scores for the Control Group,Navigation Map Group, and Both Groups Combined 201

23. Correlation Between Self-Regulation Components and Occasion 1,Occasion 2, and Improvement for Knowledge Maps for the ControlGroup 203

24. Correlation Between Self-Regulation Components and Occasion 1,Occasion 2, and Improvement for Problem Solving Retention Responsesby the Control Group 203

25. Correlation Between Self-Regulation Components and Occasion 1,Occasion 2, and Improvement for Problem Solving Transfer Responsesby the Control Group 203

26. Correlation Between Self-Regulation Components and Occasion 1,Occasion 2, and Improvement for Knowledge Maps for the ControlGroup 203

27. Correlation Between Self-Regulation Components and Occasion 1,Occasion 2, and Improvement for Problem Solving Retention Responsesby the Control Group 204

28. Correlation Between Self-Regulation Components and Occasion 1,Occasion 2, and Improvement for Problem Solving Transfer Responsesby the Control Group 204

viii


29. Correlation Between Self-Regulation Components and Occasion 1,Occasion 2, and Improvement for Knowledge Maps for Both GroupsCombined 204

30. Correlation Between Self-Regulation Components and Occasion 1,Occasion 2, and Improvement for Problem Solving Retention Responsesfor Both Groups Combined 205

31. Correlation Between Self-Regulation Components and Occasion 1,Occasion 2, and Improvement for Problem Solving Transfer Responsesfor Both Groups Combined 205

32. Descriptive Statistics of the Number of Safes Opened During Occasion 1and Occasion 2, and the Total Number of Safes Opened by the ControlGroup, Navigation Map Group, and Both Groups Combined 207

33. Means for the Number of Safes Opened by Group by Occasion 208

34. Correlation Between Self-Regulation Components and Number of SafesOpened by the Control Group 210

35. Correlation Between Self-Regulation Components and Number of SafesOpened by the Navigation Map Group 210

36. Correlation Between Self-Regulation Components and Number of SafesOpened for Both Groups Combined 210

37. Descriptive Statistics of the Continuing Motivation Scores of the Control, Navigation Map Group, and Both Groups Combined 211

ix


List of Figures

Figures Page

1. O’Neil Problem Solving Model 29

2. Knowledge Map User Interface Displaying 3 Concepts and 2 Links 73

3. Adding Concepts to the Knowledge Map 74

4. Participant Solicitation Flyer 106

5. Sample Navigation Map 108

6. Task Completion Form 1 for Pilot Study 115

7. Task Completion Form 2 for Pilot Study 115

8. Navigation Map for Game 1 120

9. Navigation Map for Game 2 120

10. Expert SafeCracker Knowledge Map 1 125



13. Sample Participant Map for the Game SafeCracker 128

14. Training Map 147

15. Task Completion Form 1 for Main Study 178

16. Task Completion Form 2 for Main Study 180

x


ABSTRACT

Cognitive load theory defines a limited capacity working memory with associated

auditory and visual/spatial channels. Navigation in computer-based hypermedia and

video game environments is believed to place a heavy cognitive load on working

memory. Current 3-dimensional (3-D) computer-based video games often include

complex, occluded environments (conditions where vision is blocked by objects in

the environment, such as internal walls, trees, hills, or buildings) preventing players

from plotting a direct visual course from a start to finish location. Navigation maps

may provide the support needed to effectively navigate in these environments.

Navigation maps are a type of graphical scaffolding, and scaffolding, including

graphical scaffolding, helps learners by reducing the amount of cognitive load placed

on working memory, thereby leaving more working memory available for learning.

Navigation maps have been shown to be effective in 3-D, occluded, video game

environments requiring complex navigation with simple problem solving tasks.

Navigation maps have also been shown to be effective in 2-dimensional

environments involving complex problem solving tasks. This study extended the

research by combining these two topics—navigation maps for navigation in 3-D,

occluded, computer-based video games and navigation maps in 2-dimensional

environments with complex problem solving tasks—by examining the effect of a

navigation map on a 3-D, occluded, computer-based video game with a complex

problem solving task. In addition to the effect of a navigation map on a problem

solving task, the effect of a navigation map on continuing motivation was examined.

Results of the study were unexpected; of the five hypotheses (four addressing

xi


problem solving outcomes and one addressing continuing motivation) only one

hypothesis was partially supported, with the other four unsupported. Two

explanations were examined, based on available data from, and components of, the

study. It is suspected that the game environment may not have been complex enough

for the treatment group to have benefited from use of a navigation map. Rather, the

navigation map may have resulted in added, unnecessary, cognitive load on the

treatment group, offsetting any cognitive benefits the navigation map was expected

to offer, thereby lowering the performance of the treatment group. The second

explanation involved strategy priming. Both the navigation map group and the

control group received a considerable and equivalent amount of problem solving

strategy priming. It is believed that this priming may have resulted in improving the

performance of both groups enough to counter any differences that might have been

observed from the treatment (the navigation map) had the priming not occurred.

Results of this study suggest that, while navigation maps have been found to be

effective for both navigation and problem solving, not all situations may require or

benefit from a navigation map. Additionally, other forms of scaffolding, such as

strategy priming, may provide a enough support to offset any gains that might be

observed from navigation map usage.

xii


CHAPTER 1

INTRODUCTION

With the current power of computers and the state-of-the-art of video

games, it is likely that future versions of educational video games will include

immersive environments in the form of three-dimensional (3-D), computer-based

games requiring navigation through occluded paths in order to perform complex

problem solving tasks. Cutmore, Hines, Maberly, Langford, and Hawgood (2000)

define immersion as a ‘view-centered perspective’ which results in “the sensation of

being situated within an environment as opposed to viewing it on a map or other

such abstract representation” (p. 223). According to Cutmore, et al., occlusion refers

to conditions where vision is blocked by objects in the environment, such as internal

walls or large environmental features like trees, hills, or buildings. Occluded paths

prevent the ability to plot a “direct visual course from the start to finish locations.

Rather, knowledge of the layout is required” (p. 224). This study examines the use of

navigation maps to support navigation through a 3-D, occluded computer-based

video game involving a complex problem solving task.

Chapter one begins with an examination of the background of the problem.

Next the purpose of the study is discussed, followed by why the study is significant

—how it will inform the literature—and the hypotheses that will be addressed. The

next sections in chapter one include an overview of the methodology that will be

utilized and a brief explanation of the organization of this dissertation.

Background of the Problem

1


Educators and trainers began to take notice of the power and potential of

computer games for education and training back in the 1970s and 1980s (Donchin,

1989; Malone, 1981; Malone & Lepper, 1987; Ramsberger, Hopwood, Hargan, &

Underfull, 1983; Ruben, 1999; Thomas & Macredie, 1994). Computer games were

hypothesized to be potentially useful for instructional purposes and were also

hypothesized to provide multiple benefits: (a) complex and diverse approaches to

learning processes and outcomes; (b) interactivity; (c) ability to address cognitive as

well as affective learning issues; and perhaps most importantly, (d) motivation for

learning (O’Neil, Baker, & Fisher, 2002).

Despite early expectations, research into the effectiveness of games and

simulations as educational media has been met with mixed reviews (de Jong & van

Joolingen, 1998; Garris, Ahlers, & Driskell, 2002). It has been suggested that the

lack of consensus can be attributed to weaknesses in instructional strategies

embedded in the media and to other issues related to cognitive load (Chalmers, 2003;

Cutmore et al., 2000; Lee, 1999; Thiagarajan, 1998; Wolfe, 1997). Cognitive load

refers to the amount of mental activity imposed on working memory at an instance in

time (Chalmers, 2003; Cooper, 1998; Sweller & Chandler, 1994, Yeung, 1999).

Researchers have proposed that working memory limitations can have an adverse

effect on learning (Sweller & Chandler, 1994; Yeung, 1999). Further, cognitive load

theory suggests that learning involves the development of schemas (Atkinson, Derry,

Renkl, & Wortham, 2000), a process constrained by limited working memory and

separate channels for auditory and visual/spatial stimuli (Brunken, Plass, & Leutner,

2


2003). Cognitive load theory also describes an unlimited capacity, long-term

memory that can store vast numbers of schemas (Mousavi, Low, & Sweller, 1995).

The inclusion of scaffolding, which provides support during schema

development by reducing the load in working memory, is a form of instructional

design; more specifically, it is an instructional strategy (Allen, 1997; Clark, 2001).

For example, graphical scaffolding, which involves the use of imagery-based aids,

has been shown to provide effective support for graphically-based learning

environments, including video games (Benbasat & Todd, 1993; Farrell & Moore,

2000-2001; Mayer, Mautone, & Prothero, 2002). Navigation maps, a particular form

of graphical scaffolding, have been shown to be an effective scaffold for navigation

of a three-dimensional (3-D) virtual environment (Cutmore et al., 2000). Navigation

maps have also been shown to be an effective support for navigating and problem

solving in a two-dimensional (2-D) hypermedia environment (Baylor, 2001; Chou,

Lin, & Sun, 2000), which is comprised of nodes of information and links between the

various nodes (Bowdish, & Lawless, 1997). What has not been examined, and is the

purpose of this study, is the effect of navigation maps utilized for navigation in a 3-

D, occluded computer-based video game on outcomes of a complex problem solving

task.

Statement of the Problem

A major instructional issue in learning by doing within simulated

environments concerns the proper type of guidance, that is, how best to create

cognitive apprenticeship (Mayer et al., 2002). A virtual environment creates a

number of issues with regards to learning. Problem solving within a virtual

3


environment involves not only the cognitive load associated with the to-be-learned

material, referred to as intrinsic cognitive load (Paas, Tuovinen, Tabbers, Van

Gerven, 2003), it also includes cognitive load related to the visual nature of the

environment, referred to as extraneous cognitive load (Brunken et al., 2003; Harp &

Mayer, 1998), as well as navigation within the environment—either germane

cognitive load or extraneous cognitive load, depending on the relationship of the

navigation to the learning task (Renkl, & Atkinson, 2003); It is germane cognitive

load if navigation is a necessary component for learning; that is, it is an instructional

strategy. It is extraneous cognitive load if navigation does not, of itself, support the

learning process; that is, it is included as a feature extraneous to content

understanding and learning. An important goal of instructional design within these

immersive environments involves determining methods for reducing the extraneous

cognitive load and/or germane cognitive load, thereby providing more working

memory capacity for intrinsic cognitive load (Brunken et al., 2003). This study will

examine the reduction of cognitive load through the use of graphical scaffolding in

the form of a navigation map, to determine if this instructional strategy can result in

better performance outcomes as reflected in retention and transfer (Paas et al., 2003)

in a game environment. Retention refers to the storage and retrieval of knowledge

and facts (Day, Arthur, & Gettman, 2001). Transfer refers to the application of

acquired knowledge and skills to new situations (Brunken et al., 2003)

Purpose of the Study

The purpose of this study is to examine the effect of a navigation map on a

complex problem solving task in a 3-D, occluded computer-based video game. The

4


environment for this study is the interior of a mansion as instantiated in the video

game SafeCracker® (Daydream Interactive, Inc., 1995/2001). The navigation map is

a printed version of the floor plan of the first floor of the mansion, with relevant

room information, such as the name of the room and the location of doors. The

problem solving task involves navigating through the environment to locate specific

rooms, to find and acquire items and information necessary to open safes located

within the prescribed rooms, and ultimately, to open the safes. With one group

playing the game while using the navigation map and the other group playing the

game without aid of a navigation map, this study will examine differences in

problem solving outcomes informed by the problem solving model defined by

O’Neil (1999); see Figure 1.

Significance of the Study

Research has examined the use of navigation maps, a particular form of

graphical scaffolding, as navigational support for complex problem solving tasks

within a hypermedia environment, where the navigation map provided an overview

of the 2-D, textual-based world which had been segmented into chunks of

information, or nodes (Chou et al., 2000). Research has also examined the use of

navigation maps as a navigational tool in 3-D virtual environments. Studies

involving 3-D environments have examined either the effect of a navigation map on

navigation within an occluded environment with the singular goal of getting from

point A to point B (Cutmore et al., 2000) or on navigation in a maze-like

environment (hallways) that included a simple problem solving task; finding a key

along the path in order to open a door at the end of the path (Galimberti, Ignazi,

5


Vercesi, &Riva, 2001). Research has not combined these two research topics; it has

not assessed the use of navigation maps in relationship to a ‘complex’ problem

solving task in a ‘complex,’ ‘occluded three-dimensional’ virtual environment.

While a number of studies on hypermedia environments have examined the

issue of 2-D maps (i.e., a site map) to aid in navigation of the various nodes for

complex problem solving tasks (e.g., Chou & Lin, 1998), no study has looked at the

effect of the use of 2-D topological maps (i.e., a floor plan) on navigation within a 3-

D video game environment in relationship to a complex problem solving task. It is

argued here that the role of the two navigation map types (2-D site map and 2-D

topological floor plan) serve the same purpose in terms of cognitive load, which is,

they reduce cognitive load by distributing some of load normally placed in working

memory to an external aid, the navigation map. In other words, information (the

structure of the environment) that would normally have been held in working

memory is offloaded to an accessible, external map of the environment. However, it

is also argued here that the spatial aspects of the two learning environments differ

substantially. A larger cognitive load is placed on the visual/spatial channel of

working memory with a 3-D video game environment as compared to a 2-D

hypermedia environment, due to the more complex visual requirements of working

within a 3-D world as opposed to a 2-D world, thereby leaving less working memory

capacity in the 3-D video game for visual stimuli; the navigation map. Therefore, the

cognitive load benefits of map usage in a 3-D environment may not be as great as the

cognitive load benefits of map usage in a 2-D environment, particularly if, as in this

6


experiment, the map is spatially separated from the main environment—the video

game—a condition which adds cognitive load (Mayer & Moreno, 1998).

As immersive 3-D video games become more widespread as commercial

entertainment, it is likely that interest will also grow for the utilization of 3-D video

games as educational media, particularly because of the perceived motivational

aspects of video games for engaging students. According to Pintrich and Schunk

(2002), motivation is “the process whereby goal-directed activity is instigated and

sustained” (p. 405). As Tennyson and Breuer (2002) contended, motivation

influences both attention and maintenance processes, generating the mental effort

that drives us to apply our knowledge and skills. Salomon (1983) described mental

effort as the depth or thoughtfulness a learner invests in processing material.

Therefore, the role of navigation maps to reduce the load induced by navigation and,

thereby, reduce burdens on working memory, is an important issue for enhancing the

effectiveness of video games as educational environments.

Research Questions and Hypotheses

Research Question 1: Will the problem solving performance of participants

who use a navigation map (the treatment group) in a 3-D, occluded computer-based

video game (i.e., SafeCracker®) be better than the problem solving performance of

those who do not use the map (the control group)?

Hypothesis 1: Participants who use a navigation map (the treatment group)

will exhibit significantly greater content understanding than participants who do not

use a navigation map (the control group).

7



will exhibit greater problem solving strategy retention than participants who do not



will exhibit greater problem solving strategy transfer than participants who do not


Hypothesis 4: There will be no significant difference in self-regulation

between the navigation map group (the treatment group) and the control group.

However, it is expected that higher levels of self-regulation will be associated with

better performance.

Research Question 2: Will the continuing motivation of participants who

use a navigation map in a 3-D, occluded computer-based video game (i.e.,

SafeCracker®) be greater than the continuing motivation of those who do not use the

map (the control group)?


will exhibit a greater amount of continuing motivation, as indicated by continued

optional game play, than participants who do not use a navigation map (the control

group).

Overview of the Methodology

This study utilized an experimental, posttest only 2x2 repeated measures

design. The first factor had 2 levels (one treatment group, and one control group).

The second factor had 2 levels (occasion 1 and occasion 2). Participants were

randomly assigned to either the treatment or the control group. Group sessions

8


involved only one group type: either all treatment participants or all control

participants. The experimental design involved administration of pretest

questionnaires, the treatment, the occasion 1 instruments, the treatment, the occasion

2 instruments, and debriefing. After debriefing, participants were offered up to 30

minutes of additional playing time (to examine continuing motivation).

Organization of the Dissertation

Chapter one provides an overview of the study with a brief introduction and

background of the topic, the problem being addressed, the significance of the study,

the hypotheses that will be tested, and an overview of the methodology of the

experiment. Chapter two is the literature review of the domains that inform the

current research: cognitive load theory, games and simulations, assessment of

problem solving, and scaffolding. Chapter three describes the study’s methodology,

with discussions of the sample, the study, the instruments, the procedures, and the

data analysis methods. Chapter four presents the results of the experiment, and

includes both descriptive and inferential statistics. Chapter five summarizes the

results and includes a discussion of the findings. Chapter six includes a summary of

the study, conclusions of the study, implications of the findings, and limitations that

may have affected the results of the study.

9


CHAPTER 2

LITERTURE REVIEW

The literature review includes information on four areas relevant to the

research topic: cognitive load theory, games and simulations, assessment of problem

solving, and scaffolding. The cognitive load section is comprised of an introduction

to cognitive load theory, including three types of cognitive load, followed by

discussions of working and long-term memory, schema development, automation,

mental models, the roles of reflection and elaboration, and metacognition. Next,

under cognitive load theory, is a discussion of meaningful learning, including the

role of mental effort, mental effort and motivation, goals and mental effort, as well as

theories related to mental effort and goal setting, self-efficacy, along with theories

and topics related to self-efficacy, and problem solving, with a discussion of the

O’Neil Problem Solving model (O’Neil, 1999). Next is a discussion of learner

control as informed by cognitive load theory. The discussion of cognitive load theory

ends with summary of the topic.

Following cognitive load theory is a discussion of games and simulations,

beginning with the defining of games, simulations, simulation-games, and video

games. Next the motivational aspects of games are introduced with a discussion of

the major characteristics of motivation: fantasy, control and manipulation, challenge

and complexity, curiosity, competition, feedback, and fun. The final section under

games and simulations is a discussion of learning and other outcomes attributed to

games and simulations. This section includes discussion of positive outcomes from

games and simulations, the relationship of motivation to negative or null outcomes

10


for games and simulations, the relationship of instructional design to learning from

games and simulations, and the roles of reflection and debriefing. The topic ends

with a summary of the games and simulations discussion.

The third section of this chapter is the assessment of problem solving

focused on the three constructs established in the O’Neil Problem Solving model

(O’Neil, 1999): measurement of content understanding, measurement of problem

solving strategies, and measurement of self-regulation. The section ends with a

summary of problem solving assessment.

The fourth and final section, scaffolding, begins with a general discussion of

scaffolding, followed by a review of the literature on a type of scaffolding relevant to

this study, graphical scaffolding. Within graphical scaffolding, research on the use

navigation maps is examined, along with the relationship of the contiguity effect and

the split attention effect to potential benefits of a navigation map. The section ends in

with a summary of scaffolding. Chapter two ends with a summary of the literature

review.

Cognitive Load Theory

Cognition is the mental faculty or process by which knowledge is acquired.

(Berube et al., 2001). Cognitive load theory, which began in the 1980s and

underwent substantial development and expansion in the 1990s (Paas et al., 2003), is

concerned with the development of instructional methods aligned with the learners’

limited cognitive processing capacity, to stimulate their ability to apply acquired

knowledge and skills to new situations (i.e., transfer). Cognitive load theory is based

on several assumptions regarding human cognitive architecture: the assumption of a

11


virtually unlimited capacity of long-term memory, schema theory of mental

representations of knowledge, and limited-processing capacity assumptions of

working memory with partly independent processing units for visual-spatial and

auditory-verbal information (Brunken et al., 2003; Mayer & Moreno, 2003; Mousavi

et al., 1995). Researchers have proposed that working memory limitations can have

an adverse effect on learning (Sweller & Chandler, 1994; Yeung, 1999).

Cognitive load is the total amount of mental activity imposed on working

memory at an instance in time (Chalmers, 2003; Cooper, 1998; Sweller & Chandler,

1994, Yeung, 1999). According to Brunken et al. (2003), cognitive load is a

theoretical construct describing the internal processes of information processing that

cannot be observed directly. Paas et al. (2003) defined cognitive load in terms of two

dimensions; an assessment dimension and a causal dimension. The above definitions

of cognitive load fit within the Paas et al.’s description of the assessment dimension,

which reflects the measurable concepts of cognitive load, mental effort, and

performance. The causal dimension reflects the interaction between task and learner

characteristics (Paas et al., 2003). This literature review will focus on the assessment

dimension and only indirectly discuss the causal dimension.

Types of Cognitive Load

Cognitive load researchers have identified up to three types of cognitive

load. All agree on intrinsic cognitive load (Brunken et al., 2003; Paas et al., 2003;

Renkl & Atkinson, 2003), which is the load involved in the process of learning; the

load required by metacognition, working memory, and long-term memory. Working

12


memory is discussed in the next section and is followed by a discussion of long-term

memory. Metacognition is discussed later under the topic of Meaningful Learning.

Another type of cognitive load agreed upon by researchers is extraneous

cognitive load. However, it is the scope of this load that is in dispute. To some

researchers, any cognitive load that is not intrinsic cognitive load is extraneous

cognitive load. To other researchers, non-intrinsic cognitive load is divided into

germane cognitive load and extraneous cognitive load. Germane cognitive load is

the cognitive load required to process the intrinsic cognitive load (Renkl, &

Atkinson, 2003). From a non-computer-based perspective, this could include

searching a book or organizing notes, in order to process the to-be-learned

information. From a computer-based perspective, this could include the interface and

controls a learner must interact with in order to be exposed to, and process, the to-be-

learned material. In contrast to germane cognitive load, these researchers see

extraneous cognitive load as the load caused by any unnecessary stimuli, such as

fancy interface designs or extraneous sounds (Brunken et al., 2003).

For each of the two working memory subsystems (visual/spatial and

auditory/verbal; see the next section “Working Memory” for further discussion of

these two subsystems), the total amount of cognitive load for a particular individual

under particular conditions can be defined as the sum of intrinsic, extraneous, and

germane cognitive loads induced by the instructional materials. Therefore, a high

cognitive load can be a result of a high intrinsic cognitive load (i.e., the nature of the

instructional content itself). It can, however, also be a result of a high germane

cognitive load (i.e., a result of activities performed on the materials that result in a

13


high memory load) or high extraneous cognitive load (i.e., a result of the inclusion of

unnecessary information or stimuli that result in a high memory load; Brunken et al.,

2003).

Each type of cognitive load (intrinsic, germane, and extraneous) is affected

by differing characteristics of the learning environment. By addressing each of these

environmental conditions, the various cognitive load types can be controlled or even

reduced. For example, the interdependence of the elements of the to-be-learned

material affects intrinsic cognitive load. According to Paas et al. (2003), low-element

interactivity refers to environments where each element can be learned

independently of the other elements, and there is little direct interaction between the

elements. High-element interactivity refers to environments where there is so much

interaction between elements that they cannot be understood until all the elements

and their interactions are processed simultaneously. As a consequence, high-element

interactivity material is difficult to understand. Element interactivity is the driver of

intrinsic cognitive load, because the demands on working memory capacity imposed

by element interactivity are intrinsic to the material being learned. Reduction in

intrinsic load can occur only by dividing the material into small learning modules

(Paas et al., 2003).

Germane cognitive load is influenced by the instructional design. The

manner in which information is presented to learners and the learning activities

required of learners are factors relevant to levels of germane cognitive load (Renkl,

& Atkinson, 2003). Renkl and Atkinson (2003) commented that, unlike extraneous

14


cognitive load which interferes with learning, germane cognitive load enhances

learning.

Extraneous cognitive load (Renkl & Atkinson, 2003) is the most controllable

load, since it is caused by materials that are unnecessary to instruction. However,

those same materials may be important for motivation. Unnecessary items are

globally referred to as extraneous. However, one category of extraneous items,

seductive details (Mayer, Heiser, & Lonn, 2001), refers to highly interesting but

unimportant elements or instructional segments. Schraw (1998) stated that these

segments usually contain information that is tangential to the main themes of a story,

but are memorable because they deal with controversial or sensational topics. The

seductive detail effect is the reduction of retention caused by the inclusion of

extraneous details (Harp & Mayer, 1998) and affects both retention and transfer

(Moreno & Mayer, 2000). By contrast, some research has proposed that learning

might benefit from the inclusion of extraneous information. Arousal theory suggests

that adding entertaining auditory adjuncts will make a learning task more interesting,

because it creates a greater level of attention so that more material is processed by

the learner (Moreno & Mayer, 2000). A possible solution to the conflict of the

seductive detail effect, which proposes that extraneous details are detrimental, and

arousal theory, which proposes that seductive details in the form of interesting

auditory adjuncts may be beneficial, is to include the seductive details, but guide the

learner away from them and to the relevant information (Harp & Mayer, 1998).

A related construct to seductive details is auditory adjuncts. According to

Banbury, Macken, Tremblay, and Jones (2001), while attempting to focus on a

15


mental activity, most of us, at one time or another, have had our attention drawn to

extraneous sounds. Banbury et al. argued that, on the surface, seductive details and

auditory adjuncts (such as sound effects or music) seem similar, but the underlying

cognitive mechanisms are different. While seductive details seem to prime

inappropriate schemas into which incoming information is assimilated, auditory

adjuncts seem to overload auditory working memory (Moreno & Mayer, 2000). For

a definition of schema, see the discussion below on schema, under Long-Term

Memory. Whether discussing intrinsic cognitive load, germane cognitive load, or

extraneous cognitive load, a major concern of research and instruction is the limits

imposed by working memory.

Working Memory

Working memory refers to the limited capacity for holding information in

mind for several seconds in the context of cognitive activity (Gevins et al., 1998). In

his seminal article, Miller (1956) described a working memory capacity of between

five and nine chunks of information. Bruning et al. (1999) defined a chunk as any

stimulus that is used, such as a letter, number, or word. Recent research has

suggested that, depending on the type of information being processed, the limited

capacity of working memory may be much lower than Miller’s (1956) findings of

five to nine chunks of information. According to Paas et al. (2003), working

memory, in which all conscious cognitive processing occurs, can handle only a very

limited number of novel interacting elements; possibly no more than two or three.

According to Baddeley (1986), working memory is comprised of three

components, central executive that coordinates two slave systems—a visuospatial

16


sketchpad for visuospatial (visual/spatial) information such as written text or

pictures, and a phonological loop for phonological (auditory/verbal) information

such as spoken text or music (Baddeley, 1986, Baddeley & Logie, 1999). All three

systems are limited in capacity and independent from one another. Load placed on

one system does not affect the load placed on the other two systems (Brunken et al.,

2003). When information enters a slave system, the information is decoded and a

mental model is constructed (see the discussion later on “Mental Models”). The

central executive system controls when the information is moved to working

memory for integration with other information, including information retrieved from

long term memory (Bruning et al., 1999). The functions of the central executive

include selecting, organizing, and integrating (Mayer, 2001). Selecting involves

attending to relevant stimuli. Organizing involves building internal connections

among the stimuli to form a coherent mental model. Integrating involves building

connections between the information (the stimuli) and prior knowledge (Mayer,

2001).

Long-Term Memory

In contrast to working memory, long-term memory has an unlimited,

permanent capacity (Tennyson & Breuer, 2002) and can contain vast numbers of

schemas (discussed below). Noyes and Garland (2003) commented that information

not held in working memory will need to be retained by the long-term memory

system. Storing more knowledge in long-term memory reduces the load on working

memory, which results in a greater capacity being made available for active

processing.

17


Schema development. Schemas are cognitive constructs that incorporate

multiple elements of information into a single element with a specific function (Paas

et al., 2003). Schema is a cognitive construct that permits people to treat multiple

sub-elements of information as a single element, categorized according to the

manner in which it will be used (Kalyuga, Chandler, & Sweller, 1998). Schemas are

generally thought of as ways of viewing the world and, in a more specific sense,

ways of incorporating instruction into our cognition. Schema acquisition is a primary

learning mechanism (Chalmers, 2003). Schemas have the functions of storing

information in long-term memory and of reducing working memory load by

permitting people to treat multiple elements of information as a single element

(Kalyuga et al., 1998; Mousavi et al., 1995). According to cognitive load theory,

multiple elements of information can be chunked as single elements in cognitive

schema (Chalmers, 2003).With schema use, a single element in working memory

might consist of a large number of lower level, interacting elements which, if

processed individually, might have exceeded the capacity of working memory (Paas

et al., 2003).

Automation. If a schema can be brought into working memory in automated

form, it will place limited demands on working memory resources, leaving more

resources available for cognitive activities such as searching for a possible problem

solution (Kalyuga et al., 1998). Controlled use of schemas requires conscious effort,

and therefore, working memory resources. By contrast, after being sufficiently

practiced over hundred of hours, schemas can operate under automatic, rather than

controlled, processing (Clark, 1999; Mousavi et al., 1995), requiring minimal

18


working memory resources and allowing for problem solving to proceed with

minimal effort (Kalyuga, Ayers, Chandler, & Sweller, 2003; Kalyuga et al., 1998;

Paas et al., 2003). Because of their cognitive benefits, the primary goals of

instruction are the construction (chunking) and automation of schemas (Paas et al.,

2003).

Mental Models

Mental models explain human understanding of external reality, translating

reality into internal representations and utilizing them in problem solving (Park &

Gittelman, 1995). According to Allen (1997), mental models are usually considered

the way in which people model processes. This emphasis on process distinguishes

mental models from other types of cognitive organizers such as schemas. A mental

model synthesizes several steps of a process and organizes them as a unit. A mental

model does not have to represent all of the steps which compose the actual process.

Mental models may be incomplete and may even be internally inconsistent (Allen,

1997). Models of mental models are termed conceptual models. Conceptual models

include: metaphor; surrogates; mapping, task-action grammars, and plans. Mental

model formation depends heavily on the conceptualizations that individuals bring to

a task (Park & Gittelman, 1995).

Elaboration and Reflection

Elaboration and reflection are processes involved to the development of

schemas and mental models. Elaborations are used to develop schemas whereby

nonarbitrary relationships are established between new information elements and the

learner’s prior knowledge (van Merrienboer, Kirshner, & Kester, 2003). According

19


to Kee and Davies (1990), elaboration consists of the creation of a semantic event

that includes the to-be-learned items in an interaction. For example, if the to-be-

learned items were the nouns ‘boat’ and ‘ocean,’ the elaboration might consist of the

creation of the semantic event “the boat crossed the ocean.”. With reflection, learners

are encouraged to consider their problem solving process and to try to identify ways

of improving it (Atkinson, Renkl, & Merrill, 2003). Reflection is reasoned and

conceptual, allowing the thinker to consider various alternatives (Howland, Laffey,

& Espinosa, 1997). According to Chi (2000), the self-explanation effect (also known

as reflection or elaboration) is a dual process that involves generating inferences and

correcting the learner’s own mental model.

Metacognition

Metacognition, or the management of cognitive processes, involves goal-

setting, strategy selection, attention, and goal checking (Jones, Farquhar, & Surry,

1995). According to Harp and Mayer (1998), many cognitive models include the

executive processes of selecting, organizing, and integrating. Selecting involves

paying attention to the relevant pieces of information. Organizing involves building

internal connections among the selected pieces of information, such as causal chains.

Integrating involves building external connections between the incoming information

and prior knowledge existing in the learner’s long-term memory (Harp & Mayer,

1998). According to Jones et al. (1995), cognitive strategies are cognitive events that

describe the way in which we process information. Metacognition is a cognitive

strategy that has executive control over other cognitive strategies. Prior experience in

solving similar tasks and using various strategies will affect the selection of a

20


cognitive strategy, such as rehearsal strategies, elaboration strategies, or organization

strategies (Jones et al., 1995).

Meaningful Learning

Meaningful learning is defined as deep understanding of the material, which

includes attending to important aspects of the presented material, mentally

organizing it into a coherent cognitive structure, and integrating it with relevant

existing knowledge (Mayer & Moreno, 2003). Meaningful learning is reflected in the

ability to apply what was taught to new situations. Meaningful learning results in an

understanding of the basic concepts of the new material through its integration with

knowledge already in long-term memory, known as the assimilation context (Davis

& Wiedenbeck, 2001).

According to assimilation theory (Ausubel, 1963, 1968), there are two kinds of

learning: rote learning and meaningful learning. Rote learning occurs through

repetition and memorization. It can lead to successful performance in situations

identical or very similar to those in which a skill was initially learned. However,

skills gained through rote learning are not easily extensible to other situations,

because they are not based on deep understanding of the material learned.

Meaningful learning, on the other hand, equips the learner for problem solving and

extension of learned concepts to situations different from the context in which the

skill was initially learned (Davis & Wiedenbeck, 2001; Mayer, 1981).

Mental Effort

Meaningful learning requires mental effort (Davis & Wiedenbeck, 2001;

Mayer, 1981). Salomon (1983) described mental effort as the depth or thoughtfulness

21


a learner invests in processing material. Mental effort is the aspect of cognitive load

that refers to the cognitive capacity that is actually allocated to accommodate the

demands imposed by a task. According to Salomon (1983), mental effort, relevant to

the task and material, appears to be the feature that distinguishes between mindless

or shallow processing from mindful or deep processing. Little effort is expended

when processing is carried out automatically or mindlessly (Salomon, 1983).

According to Clark (2003b), mental effort requires instructional messages (feedback)

that point out the novel elements of the to-be-learned material and emphasize the

need to work hard. Clark also commented that instructional messages must present

concrete and challenging, yet achievable, learning and performance goals.

Mental effort and motivation. According to Pintrich and Schunk (2002),

motivation is “the process whereby goal-directed activity is instigated and sustained”

(p. 405). Pintrich and Schunk further commented that motivation is a process that

cannot be observed directly; Instead, it is “inferred from such behaviors as choice of

tasks, persistence, and verbalizations (e.g., ‘I really want to work on this’)” (p. 5).

According to Clark (2003d), “Without motivation, even the most capable person will

not work hard” (p. 21). However, mental effort investment and motivation should not

be equated. Motivation is the driving force, but for learning to actually take place,

some specific relevant mental activity needs to be activated. This activity is assumed

to be the employment of non-automatic effortful elaborations (Salomon, 1983).

A number of variables affect motivation and mental effort. In an extensive

review of motivation theories, Eccles and Wigfield (2002) discussed Brokowski and

colleagues’ motivation model (Borkowski, Carr, Relliger, & Pressley, 1990) that

22


highlights the interaction of the following cognitive, motivational, and self-

processes: domain-specific knowledge; strategy knowledge; personal-motivational

states (including attributional beliefs, self-efficacy, and intrinsic motivation); and

knowledge of oneself (including goals and self perceptions). Each of these variables

has been examined through numerous studies. For example, in a study of college

freshmen, Livengood (1992) found that psychological variables (e.g., effort/ability,

reasoning, goal choice, and confidence) are strongly associated with academic

participation and satisfaction. And Corno and Mandinah (1983) commented that

students in classrooms actively engage in a variety of cognitive interpretations of

their environments and themselves which, in turn, influence the amount and kind of

effort they will expend on classroom tasks.

Several factors affecting motivation and mental effort will be discussed. First

will be a discussion of goals and mental effort, along with related theories. Next will

be a discussion of self-efficacy and related theories. Domain-specific knowledge will

be discussed later under the heading of Problem Solving. Self-processes and strategy

knowledge were discussed previously under the section entitled Metacognition.

Goals and mental effort. According to Clark (1999), the more novel the goal

is perceived to be, the more effort we will invest until we believe we might fail.

Clark also contended that a task should not be too easy or too hard, because in either

case, the learner will lose interest (Clark, 1999; Malone & Lepper, 1987). At the

point where a goal is perceived as too easy to be worth investment of effort, effort is

reduced as we “unchoose” the goal. At the point where failure expectations begin,

effort is reduced as we unchoose the goal to avoid a loss of control. This inverted U

23


relationship suggests that mental effort problems include two broad forms: over

confidence and under confidence (Clark, 1999). Therefore, the level of mental effort

necessary to achieve goals can be influenced by adjusting perceptions of goal

novelty and goal attainment, and the effectiveness of the strategies people use to

achieve goals (Clark, 1999).

Motivation influences both attention and maintenance processes (Tennyson

& Breuer, 2002), and generates the mental effort that drives us to apply our

knowledge and skills. As mentioned above, mental effort is goal-directed (Pintrich &

Schunk, 2002). But not all goals are motivating. For example, easy goals are not

motivating (Clark, 2003d). Further, vague goals are not as motivating as specific

goals. It has been shown that individuals given more general goals (such as “do your

best”) do not work as long as those given more specific goals, such as “list 70

contemporary authors” (Thompson et al., 2002; Locke & Latham, 2003).

Goal setting theory. According to Thompson et al. (2002), goal setting theory

is based on the simple premise that people exert effort toward accomplishing goals.

Goals may increase performance as long as a few factors are taken into account, such

as acceptance of the goal, feedback on progress toward the goal, a goal that is

appropriately challenging, and a goal that is specific (Thompson et al., 2002). Goal

setting guides the cognitive strategies in a certain direction. Goal checking are those

monitoring processes that check to see if the goal has been accomplished, or if the

selected strategy is working as expected. The monitoring process is active

throughout an activity and constantly evaluates the success of other processes. If a

24


cognitive strategy appears not to be working, an alternative may then be selected

(Jones et al., 1995).

Goal orientation theory. Goal setting theory is concerned with the prediction

that those with high performance goals and a perception of high ability will exert

great effort, and those with low ability perceptions will avoid effort (Miller et al.,

1996). Once we are committed to a goal, we must make a plan to achieve the goal. A

key element of all goal-directed planning is our personal assessment of the necessary

skills and knowledge required to achieve a goal. Related to this assessment is the

self-belief in ones ability to achieve the goal.

Self-Efficacy

Self-efficacy is a judgment of one’s ability to perform a task within a specific

domain (Bandura, 1997). A key aspect of self-efficacy assessment is our perception

of how novel and difficult the goal is to achieve. The ongoing results of this analysis

are hypothesized to determine how much effort we will invest in a goal (Clark,

1999). Perceived self-efficacy refers to subjective judgments of how well one can

execute a particular course of action, handle a situation, learn a new skill or unit of

knowledge, and the like (Salomon, 1983). Perceived self-efficacy has much to do

with how a class of stimuli is perceived. The more demanding the stimuli is

perceived to be, the less efficacious the perceiver would feel about it. Conversely,

the more familiar, easy, or shallow it is perceived, the more efficacious the perceiver

would feel about handling it. It follows that perceived self efficacy should be related

to the perception of demand characteristics (the latter includes the perceived

25


worthwhileness of expending effort), and that both should affect effort investment

jointly (Salomon, 1983).

Self-efficacy theory. According to Mayer (1998), self-efficacy theory predicts

that students work harder on a learning task when they judge themselves as capable

versus when they lack confidence in their ability to learn. Self-efficacy theory also

predicts that students understand the material better when they have high self-

efficacy than when they have low self-efficacy (Mayer, 1998). Effort is primarily

influenced by specific and detailed self efficacy assessments of the knowledge

required to achieve tasks (Clark, 1999). A person’s belief about whether he or she

has the skills required to succeed at a task is possibly the most important factor in the

quality and quantity of mental effort that a person will invest (Clark, 2003d).

Expectancy-Value Theory. Related to self-efficacy theory, expectancy-value

theories propose that the probability of behavior depends on the value of the goal

and the expectancy of obtaining that goal (Coffin & MacIntyre, 1999). Expectancies

refer to beliefs about how we will do on different tasks or activities, and values have

to do with incentives or reasons for doing the activity (Eccles & Wigfield, 2002).

From the perspective of expectancy-value theory, goal hierarchies (the importance

and the order of goals) could be organized around aspects of task value. Different

goals may be perceived as more or less useful, or more or less interesting. Eccles and

Wigfield (2002) suggested that the relative value attached to the goal should

influence its placement in a goal hierarchy, and the likelihood a person will try to

attain the goal and, therefore, exert mental effort. Clark (2003b) commented that the

more instruction supports a student’s interest and utility value for instructional goals,

26


as well as the student’s self-efficacy for a course, the more likely the student will

become actively engaged in the instruction and persist when faced with distractions.

Task value. Task value refers to an individual’s perceptions of how

interesting, important, and useful a task is (Coffin & MacIntyre, 1999). Interest in,

and perceived importance and usefulness of, a task comprise important dimensions

of task value (Bong, 2001). Citing Eccles’ expectancy-value model, Townsend and

Hicks (1997) stated that the perception of task value is affected by a number of

factors, including the intrinsic value of a task, its perceived utility value, and its

attainment value. Thus, engagement in an academic task may occur because of

interest in the task or because the task is required for advancement in some other area

(Townsend & Hicks, 1997). According to Corno and Mandinah (1983), a task linked

to one’s aspirations (a “self-relevant” task) is a key condition for task value.

Problem solving

Problem solving is the intellectual skill to propose solutions to previously

unencountered problem situations (Tennyson & Breuer, 2002). A problem exists

when a problem solver has a goal but does not know how to reach it, so problem

solving is mental activity aimed at finding a solution to a problem (Baker & Mayer,

1999). Similarly, Tennyson and Breuer (2002) stated that problem solving is

associated with situations dealing with previously unencountered problems, requiring

the integration of new information with existing knowledge to form new knowledge.

These descriptions encompass Mayer and Moreno’s (2003) definition of transfer as

the ability to apply what was taught to new situations.

27


According to Tennyson and Breuer (2002), a first condition of problem

solving involves the differentiation process of selecting knowledge that is currently

in storage using known criteria. Concurrently, this selected knowledge is integrated

to form a new knowledge. Cognitive complexity within this condition focuses on

elaborating the existing knowledge base. Problem solving may also involve

situations requiring the construction of knowledge by employing the entire cognitive

system. Therefore, the sophistication of a proposed solution is a factor of the

person’s knowledge base, level of cognitive complexity, higher-order thinking

strategies, and intelligence (Tennyson & Breuer, 2002). According to Mayer (1998),

successful problem solving depends on three components—skill, metaskill, and will

—and each of these components can be influenced by instruction. Metacognition—in

the form of metaskill—is central in problem solving because it manages and

coordinates the other components (Mayer, 1998).

O’Neil Problem Solving model. The O’Neil Problem Solving model (O’Neil,

1999, see figure 1 below) is based on Mayer and Wittrock’s (1996)

conceptualization: “Problem solving is cognitive processing directed at achieving a

goal when no solution method is obvious to the problem solver” (p. 47). This

definition is further analyzed into components suggested by the expertise literature:

content understanding or domain knowledge, domain-specific problem solving

strategies, and self-regulation (see, e.g., O’Neil, 1999, 2002). Self-regulation is

composed of metacognition (planning and self-monitoring) and motivation (effort

and self-efficacy). Thus, in the specifications for the construct of problem solving, to

be a successful problem solver, “one must know something (content knowledge),

28


possess intellectual tricks (problem solving strategies), be able to plan and monitor

one’s progress towards solving the problem (metacognition), and be motivated to

perform” (effort and self-efficacy; O’Neil, 1999, pp. 255-256).

Figure 1. O’Neil Problem Solving Model

In problem solving, the skeletal structures are instantiated in content domains,

so that a set of structurally similar models for thinking about problem solving is

applied to science, mathematics, and social studies. These models may vary in the

explicitness of problem representations, the guidance about strategy (if any), the

demands of prior knowledge, the focus on correct procedures, the focus on

convergent or divergent responses, and so on (Baker & Mayer, 1999). Domain-

specific aspects of problem solving (e.g., the part that is unique to geometry,

geology, or genealogy) involve the specific content knowledge, the specific

procedural knowledge in the domain, any domain-specific cognitive strategies (e.g.,

geometric proof, test, and fix), and domain specific discourse (O’Neil, 1998, as cited

ContentUnderstanding

Problem solvingStrategies

Self-Regulation

DomainSpecific

DomainIndependent

Metacognition Motivation

Planning Self-Monitoring

Effort Effort

Problem Solving

29


in Baker & Mayer, 1999). Both domain-independent and domain-dependent

knowledge are usually essential for problem solving. Domain-dependent analyses

focus on the subject matter as the source of all needed information (Baker & O’Neil,

2002).

Learner Control

In contrast to more traditional technologies that only deliver information,

computerized learning environments offer greater opportunities for interactivity and

learner control. These environments can offer simple sequencing and pace control or

they can allow the learner to decide which, and in what order, information will be

accessed (Barab, Young, & Wang, 1999). The term navigation refers to a process of

tracking one’s position in an environment, whether physical or virtual, to arrive at a

desired destination (Cutmore et al., 2000).

According to Cutmore et al. (2000), the route through the environment

consists of either a series of locations or a continuous movement along a path.

Effective navigation of a familiar environment depends upon a number of cognitive

factors. These include working memory for recent information, attention to

important cues for location, bearing and motion, and finally, a cognitive

representation of the environment which becomes part of a long-term memory; a

cognitive map (Cutmore et al., 2000). In this study, the control group will be subject

to the cognitive loads described by Cutmore et al. In contrast, the navigation map

provided to the treatment group will help reduce the load imposed on working

memory by aiding those participants in developing a cognitive representation of the

environment.

30


Hypermedia environments divide information into a network of multimedia

nodes, or chunks of information, connected by various links (Barab, Bowdish, &

Lawless, 1997). According to Chalmers (2003), how easily learners become

disoriented in a hypermedia environment may be a function of the user interface.

One area where disorientation can be a problem is in the use of links. Although links

create the advantage of exploration, there is always the chance learners may become

lost, not knowing where they were, where they are going, or where they have been

(Chalmers, 2003).

With regards to virtual 3-D environments, Cutmore et al. (2000) argued that

navigation becomes problematic when the whole path cannot be viewed at once and

is largely occluded by objects in the environment. Under these conditions, one

cannot simply plot a direct visual course from the start to finish locations. Rather,

knowledge of the layout of the space is required (Cutmore et al., 2000).

Daniels and Moore (2000) commented that message complexity, stimulus

features, and additional cognitive demands inherent in hypermedia, such as learner

control, may combine to exceed the cognitive resources of some learners. Dillon and

Gabbard (1998) found that novice and lower aptitude students have the greatest

difficulty with hypermedia. Children are particularly affected by the cognitive

demands of interactive computer environments. According to Howland, Laffey, and

Espinosa (1997), many educators believe that young children do not have the

cognitive capacity to interact and make sense of the symbolic representations of

computer environments.

31


In spite of the intuitive and theoretical appeal of hypertext environments,

empirical findings yield mixed results with respect to the learning benefits of learner

control over program control of instruction (Niemiec, Sikorski, & Wallberg, 1996;

Steinberg, 1989). Six extensive meta-analyses of distance and media learning studies

in the past decade have found the same negative or weak results (Bernard, et al,

2003). In reference to distance learning environments, Clark (2003c) argued that

when sequencing, contingencies, and learning strategies permit only minimal learner

control over pacing, then “except for the most advanced expert learners, learning will

be increased” (p. 14).

Summary of Cognitive Load

Cognitive Load Theory is based on the assumptions of a limited working

memory with separate channels for auditory and visual/spatial stimuli, and a virtually

unlimited capacity long-term memory that stores schemas of varying complexity and

levels of automation (Brunken et al., 2003). According to Paas et al. (2003),

cognitive load refers to the amount of load placed on working memory. Miller

(1956) found that working memory limits range from five to nine chunks of

information. Bruning et al. (1999) defined a chunk as any stimulus that is used, such

as a letter, number, or word. Recent research has suggested that working memory

may be even more limited when working with novel element; limiting its capacity to

as little as two or three novel elements (Paas et al., 2003). Cognitive load can be

reduced through effective use of the auditory and visual/spatial channels, as well as

schemas stored in long-term memory.

32


There are three types of cognitive load that can be defined in relationship to a

learning or problem solving task: intrinsic cognitive load, germane cognitive load,

and extraneous cognitive load. Intrinsic cognitive load refers to the cognitive load

placed on working memory by the to-be-learned material (Paas et al., 2003).

Germane cognitive load refers to the cognitive load required to access and process

the intrinsic cognitive load; For example, the problem solving processes that are

instantiated in the learning process so that learning can occur (Renkl & Atkinson,

2003). Extraneous cognitive load refers to the cognitive load imposed by stimuli that

neither support the learning process (i.e., germane cognitive load) nor are part of the

to-be-learned material (i.e., intrinsic cognitive load). Seductive details, a particular

type of extraneous cognitive load, are highly interesting but unimportant elements or

instructional segments that are often used to provide memorable or engaging

experiences (Mayer et al., 2001; Schraw, 1998).

An important goal of instructional design is to balance intrinsic, germane, and

extraneous cognitive loads to support learning outcomes and to recognize that the

specific balance is dependent on a number of factors (Brunken et al., 2003),

including the amount of prior knowledge and the need for motivation. Another major

factor affecting learning is element interactivity (Paas et al., 2003). Low-element

interactivity refers to environments where each element can be learned

independently of the other elements. High-element interactivity refers to

environments were there is so much interaction between elements that they cannot be

understood until all the elements and their interactions are processed simultaneously.

Element interactivity drives intrinsic cognitive load, because the demands of working

33


memory increase as element interactivity increases. Cognitive load can be reduced

by dividing the to-be-learned materials into small learning modules, thereby reducing

germane load (Paas et al., 2003).

Working memory refers to the limited capacity for holding and processing

chunks of information. According to Miller (1956) working memory capacity varies

from five to nine chunks of information. More recently, Paas et al. (2003) argued that

working memory can only handle two or three “novel” chunks of information.

Working memory is comprised of a central executive that coordinates two slave

systems: a visuospatial sketchpad for visual information and a phonological loop for

auditory information (Baddeley, 1986). All three systems are limited in capacity and

independent of one another (Brunken et al., 2003).

Long-term memory has an unlimited permanent capacity (Tennyson &

Breuer, 2002), and can contain vast amounts of schemas. Schemas are cognitive

constructs that incorporate multiple elements of information into a single element

with a specific function (Paas et al., 2003). Schemas have the functions of storing

information in long-term memory and of reducing working memory load by

permitting people to treat multiple elements of information as a single element or

chunk (Kalyuga, et al., 1998; Mousavi et al., 1995). After being sufficiently

practiced over hundred of hours, schemas can operate under automatic, rather than

controlled, processing (Clark, 1999; Mousavi et al., 1995), requiring minimal

working memory resources and allowing for problem solving to proceed with

minimal effort (Kalyuga et al., 2003; Kalyuga et al., 1998; Paas et al., 2003).

34


Because of their cognitive benefits, the primary goals of instruction are the

construction (chunking) and automation of schemas (Paas et al., 2003).

Mental models, also termed conceptual models, are internal representations

of our understanding of external reality. Mental models include: metaphor;

surrogates; mapping, task-action grammars, and plans. Because mental models

usually model processes, they differ from schema (Allen, 1997).

Elaboration and reflection are processes involved in the development of

schemas and mental models. Elaboration consists of the creation of a semantic event

that includes the to-be-learned items in an interaction (Kees & Davies, 1990).

Reflection encourages learners to consider their problem solving process and to try

to identify ways of improving it (Atkinson et al., 2003).

Metacogntion (i.e., the central executive function of working memory) is the

management of cognitive processes (Jones et al., 1995), as well as awareness of ones

own mental processes (Anderson, Krathwohl, Airasian, Cruikshank, et al., 2001).

According to Harp and Mayer (1998), many cognitive models include the executive

processes of selecting, organizing, and integrating. Selecting involves paying

attention to the relevant pieces of information. Organizing involves building internal

connections among the selected pieces of information, such as causal chains.

Integrating involves building external connections between the incoming information

and prior knowledge existing in the learner’s long-term memory (Harp & Mayer,

1998).

Meaningful learning is defined as deep understanding of the material and is

reflected in the ability to apply what was taught to new situations; i.e., problem

35


solving transfer (Mayer & Moreno, 2003). Meaningful learning requires effective

metacognitive skills: the management of cognitive processes (Jones et al., 1995),

including selecting relevant information, organizing connections among the pieces of

information, and integrating (i.e., building) external connections between incoming

information and prior knowledge that exists in long-term memory (Harp & Mayer,

1998).

Mental effort refers to the cognitive capacity allocated to a task. Mental effort

is affected by motivation, and motivation cannot exist without goals (Clark, 2003d).

Goals are further affected by a combination of self-efficacy, the belief in one’s

ability to successfully carry out a particular behavior (Davis & Wiedenbeck, 2001)

and values, which are related to the incentives or reasons for doing an activity

(Eccles & Wigfield, 2002).

Motivation and mental effort are related, yet one does not necessarily lead to

the other. According to Clark (2003d), “Without motivation, even the most capable

person will not work hard” (p. 21). But motivation does not guarantee effort.

According to Salomon (1983), while motivation is a driving force, learning will only

occur if some specific mental activity is activated in the form of non-automatic

effortful elaborations.

A number of factors affect motivation, including domain-specific knowledge,

strategy knowledge, personal-motivational states (e.g., self-efficacy and intrinsic

motivation), and knowledge of oneself (e.g., goals and self-perceptions; Browkowski

et al., 1990). Perceptions of a goal will also affect motivation. According to Clark

(1999), goals must be neither too hard nor too easy. Otherwise, motivation and

36


mental effort will drop. Goal setting theory suggests that not only must a goal be

appropriately challenging, it must be specific (Thompson et al., 2002). Goal

orientation theory proposes that those with high performance goals and high ability

will exert great effort, while those with low ability perceptions will avoid effort

(Miller et al., 1996). Related to these perceptions is self-belief in one’s ability to

achieve a goal (i.e., self-efficacy; Bandura, 1997). Self-efficacy theory predicts that

students will work harder if they judge themselves as capable of achieving a goal

versus when they lack confidence in their abilities to achieve the goal (Mayer, 1998).

Expectancy-value theory, which is related to self-efficacy theory, proposes

that the probability of behavior depends on the value of a goal and the expectancy of

attaining the goal (Coffin & MacIntyre, 1999). Different goals can be perceived as

more or less useful, or more or less interesting (Eccles & Wigfield, 2002). Task

value, which refers to how interesting, important, or useful a task is (Coffin &

MacIntyre, 1999), is affected by a number of factors, including the intrinsic value of

a task, its perceived utility value, and its attainment value. A task linked to one’s

aspirations (a “self-relevant” task) is a key condition for task value (Corno &

Mandinah, 1983).

Many tasks involve problem solving. Problem solving is “cognitive

processing directed at transforming a given situation into a desired situation when no

obvious method of solution is available to the problem solver” (Baker & Mayer,

1999, p. 272). According to Mayer (1998), successful problem solving depends on

three components—skill, metaskill, and will—and each of these components can be

influenced by instruction. Further, metacognition—in the form of metaskill—is

37


central to problem solving because it manages and coordinates the other components

(Mayer, 1998). The O’Neil Problem Solving model (O’Neil, 1999) defines three core

constructs of problem solving: content understanding, problem solving strategies,

and self-regulation. Content understanding refers to domain knowledge. Problem-

solving strategies refer to both domain-specific and domain-independent strategies.

Self-regulation is comprised of metacognition (planning and self-monitoring) and

motivation (effort and self-efficacy; O’Neil 1999, 2002).

Learner control, which is inherent in interactive computer-based media,

allows for control of pacing and sequencing (Barab, Young, & Wang, 1999). It also

provides a potential for cognitive overload in the form of disorientation; loss of place

(Chalmers, 2003). Further, Daniels and Moore (2000) argued that message

complexity, stimulus features, and additional cognitive demands inherent in

hypermedia (such as learner control) may combine to exceed the cognitive resources

of some learners. Further, learner control is a potential source for extraneous

cognitive load. Ultimately, these issues may be the cause of the mixed reviews of

learner control (Bernard, et al, 2003; Niemiec, Sikorski, & Wallberg, 1996;

Steinberg, 1989).

Games and Simulations

According to Ricci, Salas, and Cannon-Bowers (1996), “computer-based

educational games generally fall into one of two categories: simulation games and

video games. Simulation games model a process or mechanism relating task-relevant

input changes to outcomes in a simplified reality that may not have a definite

endpoint” (p. 296). Ricci et al. further comment that simulation games “often depend

38


on learners reaching conclusions through exploration of the relation between input

changes and subsequent outcomes” (p. 296). Video games, on the other hand, are

competitive interactions bound by rules to achieve specified goals that are dependent

on skill or knowledge and often involve chance and imaginary settings (Randel,

Morris, Wetzel, & Whitehill, 1992).

One of the first problems areas with research into games and simulations is

misuse of terminology. Many studies that claim to have examined the use of games

did not use a game (e.g., Santos, 2002). At best, Santos (2002) used an interactive

multimedia that exhibited some of the features of a game, but not enough features to

actually be called a game. A similar problem occurs with simulations. A large

number of research studies use simulations but call them games (e.g., Mayer et al.,

2002). Because the goals and features of games and simulations differ, it is important

when examining the potential effects of the two media to be clear about which one is

being examined. However, there is little consensus in the education and training

literature on how games and simulations are defined.

Games

According to Garris et al. (2002), early work in defining games suggested that

there are no properties that are common to all games and that games belong to the

same semantic category only because they bear a family resemblance to one another.

Betz (1995-1996) argued that a game is being played when the actions of individuals

are determined by both their own actions and the actions of one or more actors.

A number of researchers agree that games have rules (Crookall, Oxford, &

Saunders, 1987; Dempsey, Haynes, Lucassen, & Casey, 2002; Garris et al., 2002;

39


Ricci, 1994). Researchers also agree that games have goals and strategies to achieve

those goals (Crookall & Arai, 1995; Crookall et al., 1987; Garris et al., 2002; Ricci,

1994). Many researchers also agree that games have competition (e.g., Dempsey et

al., 2002) and consequences such as winning or losing (Crookall et al., 1987;

Dempsey et al., 2002).

Betz (1995-1996) further argued that games simulate whole systems, not

parts, forcing players to organize and integrate many skills. Students will learn from

whole systems by their individual actions; individual action being the student’s game

moves. Crookall et al. (1987) also noted that a game does not intend to represent any

real-world system; it is a “real” system in its own right. According to Duke (1995),

games are situation specific. If well designed for a specific situation or condition, the

same game should not be expected to perform well in a different environment.

Simulations

In contrast to games, Crookall and Saunders (1989) viewed a simulation as a

representation of some real-world system that can also take on some aspects of

reality. Similarly, Garris et al. (2002) wrote that key features of simulations are they

represent real-world systems. However, Henderson, Klemes, and Eshet (2000)

commented that a simulation attempts to faithfully mimic an imaginary OR real

environment that cannot be experienced directly, for such reasons as cost, danger,

accessibility, or time. Berson (1996) also argued that simulations allow access to

activities that would otherwise be too expensive, dangerous, or impractical for a

classroom. Lee (1999) added that a simulation is defined as a computer program that

relates elements together through cause and effect relationships.

40


Thiagarajan (1998) argued that simulations do not reflect reality; they reflect

someone’s model of reality. According to Thiagarajan, a simulation is a

representation of the features and behaviors of one system through the use of

another. At the risk of introducing a bit more ambiguity, Garris et al. (2002)

proposed that simulations can contain game features, which leads to the final

definition: simulation-games.

Simulation-Games

Combining the features of the two media, games and simulations, Rosenorn

and Kofoed (1998) described simulation/gaming as a learning environment where

participants are actively involved in experiments, for example, in the form of role-

plays, or simulations of daily work situations, or developmental scenarios. This

paper will use the definitions of games, simulations, and simulation-games as

defined by Gredler (1996), which combine the most common features cited by the

various researchers, and yet provide clear distinctions between the three media.

Games, Simulations, and Simulation-Games

According to Gredler,

Games consist of rules that describe allowable player moves, game constraints and privileges (such as ways of earning extra turns), and penalties for illegal (nonpermissable) actions. Further, the rules may be imaginative in that they need not relate to real-world events (p. 523).

This definition is in contrast to a simulation, which Gredler (1996) defines as

“a dynamic set of relationships among several variables that (1) change over time

and (2) reflect authentic causal processes” (p. 523). In addition, Gredler describes

games as linear and simulations as non-linear, and games as having a goal of

41


winning while simulations have a goal of discovering causal relationships. Gredler

also defines a mixed metaphor referred to as simulation games or gaming

simulations, which is any blend of the features of the two interactive media: games

and simulations. Table 1 summarizes the characteristics of games, and simulations,

including two characteristics proposed by this researcher; linear goal structure and

linear intervention.

When Gredler describes games as linear and simulations as non-linear, and

describes games as having a goal of winning while simulations have a goal of

discovering causal relationships, she is linking those two characteristics. In other

words, she is stating that games have linear goal structures and simulations have

non-linear goal structures. For example, the goal of a game might be to destroy a

cannon. Then, once the cannon is destroyed, the next goal will be to storm the

fortress. Then, once the fortress is secured, the next goal might be to locate the

enemy’s plans for invasion. This linear structure is a typical format for games—do

this, then do that, then do something else. And if the player wanted to try a different

approach to, for example, destroying the cannon, he or she would have to restart the

game, or restart the level, or load a previously saved game state. Games are not

designed to allow goals to be repeated. With simulations, however, the typical goal is

to examine causal relationships.

The order in which that discovery occurs in a simulation is typically up to the

user. And once the goal is reach, the experimenter may continue by examining other

possibilities for achieving the same goal, or the experimenter can begin working

toward a new goal. Unlike players in a game, if users of a simulation wish to

42


examine an alternative approach to achieving a goal, they do not have to restart the

simulation or a level. They simply alter the input variables and observe the outcome.

Therefore, as stated, games have linear goal structures and simulations have non-

linear goal structures.

This researcher contends there is another characteristic of games and

simulations that involves either linearity or non-linearity, and that is the medium’s

intervention structure. For both games and simulations, this intervention structure is

non-linear. Intervention refers to the actions a player or user are allowed take at any

given movement of the game or simulation. In almost all instances of intervention,

both media give a least two choices. In a game, that might be to save or quit, pick up

a gun or not, open a door or back away from the door, turn left or turn right. In a

simulation, the user might have the choice to save or quit, increase one variable

value’s or decrease it, introduce another variable or remove a variable, etc.

Therefore, for both games and simulations, the intervention structure is non-linear. In

Table 1, the characteristics of simulation-games are not included, since they are

comprised of any combination of games and simulations and, therefore, are implied

in the Table.

43


Table 1Characteristics of Games and Simulations

Characteristic Game SimulationCombination of ones actions

plus at least one other’s actions

Yes (via human or computer)

Yes

Rules Defined by game designer/developer

Defined by system being replicated

Goals To win To discover cause-effect relationships

Requires strategies to achieve goals

Yes Yes

Includes competition Against computer or other players

No

Includes chance Yes YesHas consequences Yes (e.g., win/lose) YesSystem size Whole Whole or PartReality or Fantasy Both BothSituation Specific Yes YesRepresents a prohibitive

environment (due to cost, danger, or logistics)

Yes Yes

Represents authentic cause-effect relationships

No Yes

Requires user to reach own conclusion

Yes Yes

May not have definite end point

No Yes

Contains constraints, privileges, and penalties (e.g. earn extra moves, lose turn)

Yes No

Linear goal structure Yes NoLinear intervention No NoIs intended to be playful Yes No

Video Games

Just are there are disagreements for the terms game and simulation, there is

disagreement with the term video game. According to Novak (2005), the term video

44


game came from the arcade business and gravitated to the home console business;

Consoles include the Microsoft Xbox and the Sony Playstation II. Novak contended

that games played on personal computers are computer games, not video games.

However, Soanes (2003) defined a video game as “a game played by electronically

manipulating images produced by a computer program.” This would classify

computer-based games a video game. Similarly, the American Heritage Dictionary

of the English Language, fourth edition (2000), defined a video game as “an

electronic or computerized game played by manipulating images on a video display

or television screen.”

While many researcher have referred to computer-based games as video

games (e.g., Greenfield et al., 1994, 1996; Okagaki & Frensch, 1994), others have

referred to computer-based games as computer games or computer-based games

(e.g., Baker et al., 1993; Gopher et al., 1994; Hong & Liu, 2003; Williams &

Clippinger, 2002). According to Kirriemuir (2002b), the terms video game and

computer game are often used interchangeably. In this document we will use the

terms video game, computer game, and computer-based game interchangeably.

Motivational Aspects of Games

According to Garris et al. (2002), motivated learners are easy to describe;

they are enthusiastic, focused and engaged, interested in and enjoy what they are

doing, and they try hard and persist over time. Furthermore, they are self-determined

and driven by their own volition rather than external forces (Garris et al., 2002).

Ricci et al. (1996) defined motivation as “the direction, intensity, and persistence of

attentional effort invested by the trainee toward training” (p. 297). According to

45


Malouf (1987-1988), continuing motivation is defined as returning to a task or a

behavior without apparent external pressure to do so when other appealing behaviors

are available. Similarly, Story and Sullivan (1986) commented that the most

common measure of continuing motivation is whether a student returns to the same

task at a later time. A construct similar to Story and Sullivan’s definition of

continuing motivation is persistence, which is defined by Pintrich and Schunk (2002)

as “…the continuation of behavior until the goal is obtained and the need is reduced”

(p. 30).

With regard to video games, Asakawa and Gilbert (2003) argued that, without

sources of motivation, players often lose interest and drop out of a game. However,

there seems little agreement among researchers as to what those sources are—the

specific set of elements or characteristics that lead to motivation in any learning

environment, and particularly with educational games. According to Rieber (1996)

and McGrenere (1996), motivational researchers have offered the following

characteristics as common to all intrinsically motivating learning environments:

challenge, curiosity, fantasy, and control (Davis & Wiedenbeck, 2001; Lepper &

Malone, 1987; Malone, 1981; Malone & Lepper, 1987). Malone (1981) and others

also included fun as a criteria for motivation.

Stewart (1997) added the motivational importance of goals and outcomes.

Locke and Latham (1990) also commented on the robust findings with regards to

goals and performance outcomes. Locke and Latham argued that clear, specific goals

allow the individual to perceive goal-feedback discrepancies, which are seen as

crucial in triggering greater attention and motivation. Clark (2001) further argued

46


that motivation cannot exist without goals. Feedback is the final construct cited in

this dissertation as affecting attitudes (Ricci et al., 1996). Feedback is also related to

goals; Clark (2003) commented that, for feedback to be effective, it must be based on

clearly understood, concrete goals.

The following sections will focus on fantasy, control and manipulation,

challenge and complexity, curiosity, competition, feedback, and fun. The role of

goals in fostering effort and motivation was discussed earlier in this dissertation.

Fantasy

Research suggests that material may be learned more readily when presented

in an imagined context that interests the learner than when presented in a generic or

decontextualized form (Garris et al., 2002). Malone and Lepper (1987) defined

fantasy as an environment that evokes “mental images of physical or social situations

that do not exist” (p. 250). Rieber (1996) commented that fantasy is used to

encourage learners to imagine they are completing an activity in a context in which

they are really not present. However, Rieber described two types of fantasies:

endogenous and exogenous. Endogenous fantasy weaves relevant fantasy into a

game, while exogenous fantasy simply sugar coats a learning environment with

fantasy. An example of an endogenous fantasy would be the use of a laboratory

environment to learn chemistry, since this environment is consistent with the

domain. An example of an exogenous environment would be to use a hangman game

to learn spelling, because hanging a person has nothing to do with spelling. Rieber

(1996) noted that endogenous fantasy, not exogenous fantasy, is important to

intrinsic motivation, yet exogenous fantasies are a common and popular element of

47


many educational games. Intrinsic motivation is defined by Pintrich and Schunk

(2002) as “…motivation to engage in an activity for its own sake” (p. 245).

According to Malone and Lepper (1987), fantasies can offer analogies or

metaphors for real-world processes that allow the user to experience phenomena

from varied perspectives. A number of researchers (Anderson & Pickett, 1978;

Ausubal, 1963; Malone & Lepper, 1978, 1987; Singer, 1973) argued that fantasies in

the form of metaphors and analogies provide learners with better understanding by

allowing them to relate new information to existing knowledge. According to Davis

and Wiedenbeck (2001), metaphor also helps learners to feel directly involved with

objects in the domain so the computer and interface becomes invisible.

Control and Manipulation

Hannifin and Sullivan (1996) define control as the exercise of authority or the

ability to regulate, direct, or command something. Control, or self-determination,

promotes intrinsic motivation because learners are given a sense of control over the

choices of actions they may take (deCharms, 1986; Deci, 1975; Lepper & Greene,

1978). Furthermore, control implies that outcomes depend on learners’ choices and,

therefore, learners should be able to produce significant effects through their own

actions (Davis & Wiedenbeck, 2001). According to Garris et al. (2002), games evoke

a sense of personal control when users are allowed to select strategies, manage the

direction of activities, and make decisions that directly affect outcomes, even if those

actions are not instructionally relevant.

However, Hannafin & Sullivan (1996) warned that research comparing the

effects of instructional programs that control all elements of the instruction (program

48


control) and instructional programs in which the learner has control over elements of

the instructional program (learner control) on learning achievement has yielded

mixed results. Dillon and Gabbard (1998) commented that novice and lower aptitude

students have greater difficulty when given control, compared to experts and higher

aptitude students; Niemiec et al. (1996) argued that control does not appear to offer

any special benefits for any type of learning or under any type of condition.

Challenge and complexity

Challenge, is defined as “to arouse or stimulate especially by presenting with

difficulties” (retrieved from Webster’s Online Dictionary, June, 8, 2005,

http://www.m-w.com/cgi-bin/dictionary?book=Dictionary&va=challenge). Berube

(2001) defined challenge as a “requirement for full use of one’s abilities or

resources” (p. 185). These two definitions embody the idea that challenge is related

to intrinsic motivation and occurs when there is a match between a task and the

learner’s skills. The task should not be too easy nor too hard, because in either case,

the learner will lose interest (Clark, 1999; Malone & Lepper, 1987). Clark (1999)

describes this effect as an inverted U-shaped relationship with lack of effort existing

on either side of difficultly ranging from too easy to too hard. Stewart (1997)

similarly commented that games that are too easy will be dismissed quickly.

According to Garris et al. (2002), there are several ways in which an optimal

level of challenge can be obtained. Goals should be clearly specified, yet the

probability of obtaining that goal should be uncertain, and goals must also be

meaningful to the individual. Garris and colleagues argued that linking activities to

valued personal competencies, embedding activities within absorbing fantasy

49


scenarios, or engaging competitive or cooperative motivations could serve to make

goals meaningful. This relationship of meaningful goals, belief in the probability of

goal attainment, and valued personal competencies are the components of the

expectancy-value theory which suggests that the probability of behavior, in this case

motivated behavior, depends on the value of the goal and the expectancy of

obtaining that goal (Coffin & MacIntyre, 1999).

Curiosity

According to Rieber (1996), challenge and curiosity are intertwined.

Curiosity arises from situations in which there is complexity, incongruity, and

discrepancy (Davis & Wiedenbeck, 2001). Sensory curiosity is the interest evoked by

novel situations and cognitive curiosity is evoked by the desire for knowledge

(Garris et al. 2002). Cognitive curiosity motivates the learner to attempt to resolve

the inconsistency through exploration (Davis, & Wiedenbeck, 2001). Curiosity is

identified in games by unusual visual or auditory effects and by paradoxes,

incompleteness, and potential simplifications (Westbrook & Braithwaite, 2002).

Curiosity is the desire to acquire more information, which is a primary component of

the players’ motivation to learn how to operate a game (Westbrook & Braithwaite,

2001).

Malone and Lepper (1987) noted that curiosity is one of the primary factors

that drive learning and is related to the concept of mystery. Garris et al. (2002)

commented that curiosity is internal, residing in the individual, and mystery is an

external feature of the game itself. Thus, mystery evokes curiosity in the individual,

and this leads to the question of what constitutes mystery (Garris et al. 2002).

50


Research suggests that mystery is enhanced by incongruity of information,

complexity, novelty, surprise, and violation of expectations (Berlyne, 1960),

incompatibility between ideas and inability to predict the future (Kagan, 1972), and

information that is incomplete and inconsistent (Malone & Lepper, 1987).

Competition

Studies on competition with games and simulations have mixed results, due to

preferences and reward structures. A study by Porter, Bird, and Wunder (1990-1991)

examining competition and reward structures found that the greatest effects of

reward structure were seen in the performance of those with the most pronounced

attitudes toward either competition or cooperation. The results also suggested that

performance was better when the reward structure matched the individual’s

preference. According to the authors, implications of their study are that emphasis on

competition will enhance the performance of some learners but will inhibit the

performance of others (Porter et al., 1990-1991).

Yu (2001) investigated the relative effectiveness of cooperation with and

without inter-group competition in promoting student performance, attitudes, and

perceptions toward subject matter studied, computers, and interpersonal context.

With fifth-graders as participants, Yu found that cooperation without inter-group

competition resulted in better attitudes toward the subject matter studies and

promoted more positive inter-personal relationships both within and among the

learning groups, as compared to competition (Yu, 2001). The exchange of ideas and

information both within and among the learning groups also tended to be more

51


effective and efficient when cooperation did not take place in the context of inter-

group competition (Yu, 2001).

Feedback

Feedback within games can be provided for learners to quickly evaluate their

progress against the established game goal. This feedback can take many forms, such

as textual, visual, and aural (Rieber, 1996). According to Ricci et al. (1996), within

the computer-based game environment, feedback is provided in various forms

including audio cues, score, and remediation immediately following performance

(e.g., after-action review). The researchers argued that these feedback attributes can

produce significant differences in learner attitudes, resulting in increased attention to

the learning environment. Clark (2003) argued that, for feedback to be effective, it

must be based on “concrete learning goals that are clearly understood” (p. 18) and

that it should describe the gap between the learner’s current performance and the

goal. Additionally, the feedback must not be focused on the failure to achieve the

goal (Clark, 2003).

Fun

Quinn (1994, 1997) argued that for games to benefit educational practice and

learning, they need to combine fun elements with aspects of instructional design and

system design that include motivational, learning, and interactive components.

According to Malone (1981) three elements (fantasy, curiosity, and challenge)

contribute to the fun in games. While fun has been cited as important for motivation

and, ultimately, for learning, there is little empirical evidence supporting the concept

of fun. It is possible that fun is not a construct but, rather, represents an amalgam of

52


other concepts or constructs. Relevant alternative concepts or constructs include

play, engagement, and flow.

Play. Resnick and Sherer (1994) defined play as entertainment without fear of

present or future consequences; it is fun. According to Rieber, Smith, and Noah

(1998), serious play describes an intense learning experience in which both adults

and children voluntarily devote enormous amounts of time, energy, and commitment

and, at the same time, derive great enjoyment from the experience. Webster et al.

(1993) found that labeling software training as play showed improved motivation

and performance.

Flow. Csikszentmihalyi (1975; 1990) defines flow or a flow experience as an

optimal experience in which a person is so involved in an activity that nothing else

seems to matter. When completely absorbed in an activity, a person is ‘carried by the

flow,’ hence the origin of the theory’s name (Rieber & Matzko, 2001). Rieber and

Matzko (2001) offered a broader definition of flow, commenting that a person may

be considered in flow during an activity when experiencing one or more of the

following characteristics: Hours pass with little notice; challenge is optimized;

feelings of self-consciousness disappear; the activity’s goals and feedback are clear;

attention is completely absorbed in the activity; one feels in control; and one feels

freed from other worries (Rieber & Matzko, 2001). According to Davis and

Wiedenbeck (2001), an activity that is highly intrinsically motivating can become

all-encompassing to the extent that the individual experiences a sense of total

involvement, losing track of time, space, and other events. Davis and Wiedenbeck

also argued that the interaction style of a software package is expected to have a

53


significant effect on intensity of flow. It should be noted that Reiber and Matzko

commented that play and flow differ in one respect; learning is an expressed

outcome of serious play but not of flow.

Engagement. Davis and Wiedenback (2001) defined engagement as a feeling

of directly working on the objects of interest in the virtual world rather than on

surrogates. According to Davis and Wiedenbeck, this interaction or engagement can

be used along with the components of Malone and Lepper’s (1987) intrinsic

motivation model to explain the effect of an interaction style on intrinsic motivation,

or flow. Garris et al. (2002) commented that training professionals are interested in

the intensity of involvement and engagement that computer games can invoke, to

harness the motivational properties of computer games to enhance learning and

accomplish instructional objectives.

Learning and Other Outcomes for Games and Simulations

Results from studies reporting on the performance and learning outcomes

from games are mixed. This section is subdivided into four discussions. First will be

a discussion of studies indicating positive results regarding performance and learning

outcomes attributed to games and simulations; both empirical and non-empirical

studies will be discussed. Second will be a discussion of studies indicating a link

between motivation and negative or null results regarding performance and learning

outcomes attributed to games and simulations. Third will be a discussion of the

relationship of instructional design to effectiveness of educational games and

simulations as an explanation of the mixed findings among game and simulation

studies. Last will be a discussion of reflection and debriefing as a necessary

54


component to learning, with specific references to the learning instantiated in games

and simulations.

Positive Outcomes from Games and Simulations

Simulations and games have been cited as beneficial for a number of

disciplines and for a number of educational and training situations, including

aviation training (Salas, Bowers, & Rhodenizer, 1998), aviation crew resource

management (Baker, Prince, Shrestha, Oser, & Salas, 1993), laboratory simulation

(Betz, 1995-1996), chemistry and physics education (Khoo & Koh, 1998), urban

geography and planning (Adams, 1998; Betz, 1995-1996), farm and ranch

management (Cross, 1993), language training (Hubbard, 1991), disaster management

(Stolk, Alexandrian, Gros, & Paggio, 2001), and medicine and health care

(Westbrook & Braithwaite, 2001; Yair, Mintz, & Litvak, 2001). For business, games

and simulations have been cited as useful for teaching strategic planning (Washburn

& Gosen, 2001; Wolfe & Roge, 1997), finance (Santos, 2002), portfolio

management (Brozik & Zapalska, 2002), marketing (Washburn & Gosen, 2001),

knowledge management (Leemkuil, de Jong, de Hoog, & Christoph, 2003), and

media buying (King & Morrison, 1998).

In addition to teaching domain-specific skills, games have been used to

impart more generalizable skills. Since the mid 1980s, a number of researchers have

used the game Space Fortress, a 2-D, simplistic arcade-style game, with a hexagonal

“fortress” in the center of the screen surrounded by two concentric hexagons, and a

space ship, to improve spatial and motor skills that transfer far outside gameplay,

such as significantly improving the results of fighter pilot training (Day et al., 2001).

55


Also, in a series of five experiments, Green and Bavelier (2003) showed the potential

of video games to significantly alter visual selection attention. Similarly, Greenfield,

DeWinstanley, Kilpatrick, & Kaye (1994) found, with experiments involving college

students, that video game practice could significantly alter the participants’ strategies

of spatial attentional deployment—speed in which a participant would find and

respond to a visual stimulus on a target display.

Of the various articles discussed in the last two paragraphs, all studies in the

first paragraph were non-empirical. All studies in the second paragraph were

empirically based—the studies by Day et al. (2001), Green and Bavelier (2003), and

Greenfield et al. (1994). Table 2 shows the medium, the measure, and the participant

age for all the articles referenced in the first paragraph and only the Greenfield et al.

article referenced in the second paragraph (the other two appear in Table 3, which is

discussed immediately after Table 2). With the exception of the Greenfield et al.

study, all articles in Table 2 are non-empirical studies and contain three primary

shortcomings. First, they did not include a control group. Second, the primary source

for data was self-report. And third, they did not assess learning outcomes, except

through self-report of perceived learning. For example, Santos (2002) commented

that the survey in his study did not capture was the degree to which students actually

learned as a result of participating in the game. He further commented that students

may have enjoy participating in the computer-based exercises and may report

learning, but to demonstrate that media actually facilitates learning is difficult for

researchers (Santos, 2002).

56


Table 2Non-empirical Studies: Media, Measures, and Participants

Study Mediaa Measures Participant Ageb

Adams (1998) SimCity 2000 (b) Survey on media preference, perceptions, SimCity prior experience, results of experiments. No control group

Adult

Baker, Prince, Shrestha, Oser, & Salas (1993)

Microsoft Flight Simulator (b)

Observation. Reaction survey. No control group.

Adult

Betz (1995-1996) SimCity 2000 (b) Content understanding exam. Perception and attitude survey. No control group.

Adult

Brozik & Zapalska (2002)

The Portfolio Game (b)

Simulation performance; not knowledge gains. No control group.

Adult

Cross (1993) AgVenture (c) Reaction and self-assessment survey. No control group

Adult

Greenfield, DeWinstanley, Kilpatrick, & Kaye (1994)

Robot Battle (a) Performance after game on visual attention task. Treatment/control based on prior gaming experience.

Adult

Hubbard (1991) Hangman (c) Discussion of games and language learning

NotApplicable

a Letters in parentheses indicate type of media: a = game; b= simulation; c= simulation game.b Participants below college are defined as child. Participants college age or higher are defined as adult.

57


Table 2 (Continued)Non-empirical Studies: Media, Measures, and Participants


Khoo & Koh (1998)

Cerius2 (b) Self-assessment questionnaire on content understanding. No control group.

Adult

King & Morrison (1998)

Media Buying Simulation (b)

Perceived learning outcomes and value of simulation.

Adult

Leemkuil, de Jong, de Hoog, & Christoph (2003)

KM Quest (web-based) (c)

Formative evaluation of application functionality

Adult

Salas, Bowers, & Rhodenizer (1998)

Various Simulations (b)

Discussion of various studies

N/A

Santos (2002) Financial system simulator (b)

Self-assessment survey on perceived content-understanding and motivation. No control group

Adult

Stolk, Alexandrian, Gros, & Paggio (2001)

Web-based disaster management multimedia (c)

Formative evaluation of medium.

Adults

Washburn & Gosen (2001)

Micromatic (c) Pre-post knowledge gains. Attitude survey. No control group.


58


Table 2 (Continued)Non-empirical Studies: Media, Measures, and Participants


Westbrook & Braithwaite (2001)

Health Care Game (c) Pre- and Post-self-assessment questionnaires: domain knowledge; attitude toward working in groups. Exam to assess factual knowledge gains. No control group.

Adults

Wolfe & Roge (1997)

Various simulations and simulation games (b, c)

Discuss of games that teach strategic management.

Adults

Yair, Mintz, & Litvak (2001)

Touch the Sky, Touch the Universe (b)

Discussion of the simulation

N/A

a Letters in parentheses indicate type of media: a = game; b = simulation; c = simulation game.b Participants below college are defined as child. Participants college age or higher are defined as adult.

A recent review of empirically based studies on the use of games and

simulations for teaching or training adults over the last 15 years was conducted by

O’Neil and Wainess (in preparation). Of the thousands of journal articles published

on games and simulations over the past 15 years (including those listed in Table 2)

by using the search terms games, computer game, PC game, computer video game,

video game, cooperation game, and multi-player game, only 18 empirically-based

journal articles were found with either qualitative or quantitative information on the

effectiveness of games with adults as participants. Research based on dissertations or

technical reports was not examined. A hand search of journals for 2004/2005 found

one additional journal article that met the search criteria, for a total of 19 journal

59


articles. Table 3 shows the medium, the measure, and the participant age for all the

empirical studies found by the search.

Table 3Empirical Studies: Media, Measures, and Participants


Arthur et al. (1995)

Space Fortress (a) Performance on game, visual attention

Adult

Carr & Groves (1998)

Business Simulation in Manufacturing Management (c)

Survey Adult

Day, Arthur, & Gettman (2001)

Space Fortress (a) Performance on game and knowledge map

Adult

Galimberti, Ignazi, Vercesi, & Riva (2001)

3D-Maze (a) Observation and time to complete game

Adult

Gopher, Weil, & Baraket (1994)

Space Fortress II (a) Performance on game and flight performance

Adult

Green & Bavelier (2003)

Medal of Honor (c) Visual attention Adult

Green & Flowers (2003)

Video catching task (c)

Performance on game,

exit questionnaire

Adult

Mayer, Mautone, & Prothero (2002)

Profile Game (c) Performance on retention and transfer tests

Adult

Moreno & Mayer (2000b)

Design-a-Plant (b) Performance on retention and transfer tests, plus survey

Adult

Morris, Hancock, & Shirkey (2004)

Delta Force (c) Performance on game, Stress questionnaire, observation of military tactics used

Adult

Parchman, Ellis, Christinaz, & Vogel (2002)

Adventure Game (a) Retention test, transfer test, motivation questionnaire

Adult


60


Table 3 (Continued)Empirical Studies: Media, Measures, and Participants


Porter, Bird, & Wunder (1990-1991)

Whale Game (a) Performance on game, satisfaction survey

Adult

Prislin, Jordan, Worchel, Senmmer, & Shebilske (1996)

Space Fortress (a) Performance on game, observation, discussion behavior

Adult

Rhodenizer, Bowers, & Bergondy (1998)

AIRTANDEM (b) Performance on game, retention tests

Adult

Ricci, Salas, & Cannon-Bowers (1996)

QuizShell (b) Performance on pre-, post-, and retention tests, and trainee reaction questionnaire

Adult

Rosenorn & Kofoed (1998)

Experiment Atrium (b)

Observation Adult

Shebilske, Regian,

Arthur, & Jordan (1992)

Space Fortress (a) Performance on game

Adult

Shewokis (2003) Winter Challenge Performance on game

Adult

Tkacz (1998) Maze game (c) Performance on game, transfer test of position location

Adult


Many studies claiming positive outcomes appear to be making unsupported

claims for the media. This was particularly true for the non-empirical studies listed in

61


Table 2. While less often in the empirical studies listed in Table 3, there were

instances where outcome claims appeared to be substantiated. For example, Mayer,

Mautone, and Prothero (2002) examined performance outcomes using retention and

transfer tests, and Carr and Groves (1998) examined performance outcomes using

self-report surveys. The Mayer et al. (2002) study offered strong statistical support

for their findings, using retention and transfer tests, whereas Carr and Groves used

only participants’ self-reports as evidence of learning effectiveness. In Carr and

Groves’ study, participants reported their belief that they learned something from the

experience. No cognitive performance was actually measured, yet Carr and Groves

suggested that their simulation game was a useful educational tool, and that use of

the tool provided a valuable learning experience. It should also be noted that, while

the Mayer et al. study included both treatment and control groups, the Carr and

Groves study involved only treatment groups. As exemplified by the unsubstantiated

claims of Carr and Groves (1998), Leemkuil et al. (2003) commented that much of

the work on the evaluation of games has been anecdotal, descriptive, or judgmental.

A further complication, as discussed earlier in this document, is the issue of

mislabeling media. For example, in their study involving three forms of Chemical,

Biological, and Radiological Defense (CBRD) training for Naval recruits, including

use of a game, Ricci et al. (1996) claimed that results of their study provided

evidence that computer-based gaming can enhance learning and retention of

knowledge. However, the medium used in their study met the criteria for a

simulation game, not a game. Therefore, their claim should have promoted the

benefits of a simulation game, not a game.

62


Relationship of Motivation to Negative or Null Outcomes from Games and

Simulations

A number of researchers have addressed the issue of the motivational aspects

of games, arguing that the motivation attributed to enjoyment of educational games

may not necessarily indicate learning and, possibly, might indicate less learning.

Garris et al. (2002) noted that, although students generally seem to prefer games over

other, more traditional, classroom training media, reviews have reported mixed

results regarding the training effectiveness of games.

Druckman (1995) concluded that games seem to be effective in enhancing

motivation and increasing student interest in subject matter, yet the extent to which

that translates into more effective learning is less clear. As a note of caution,

Brougere (1999) commented that anything that contributes to the increase of emotion

(such as the quality of the design of video games) reinforces the attraction of the

game but not necessarily its educational effectiveness. Similarly, Salas et al. (1998)

commented that liking a simulation does not necessarily transfer to learning.

Salomon (1984) went even further, by commenting that a more positive attitude can

actually indicate less learning. And in an early meta-analysis of the effectiveness of

simulation games, Dekkers and Donatti (1981) found a negative relationship between

duration of training and training effectiveness. Simulation games became less

effective the longer the game was played (suggesting that perhaps trainees became

bored over time). Clark and Sugrue (2001) described a novelty effect where student

effort and attention is high when a medium is novel (new) but which tends to

diminish over time as students become more familiar with the medium.

63


Relationship of Instructional Design to Learning from Games and Simulations

de Jong and van Joolingen (1998), after reviewing a large number of studies

on learning from simulations, concluded, “There is no clear and univocal outcome in

favor of simulations. An explanation of why simulation based learning does not

improve learning results can be found in the intrinsic problems that learners may

have with discovery learning” (p. 181). These problems are related to processes such

as hypothesis generation, design of experiments, interpretation of data, and

regulation of learning. After analyzing a large number of studies, de Jong and van

Joolingen (1998) concluded that adding instructional support to simulations might

help to improve the situation.

The hypothesis is that games themselves are not sufficient for learning but

there are elements in games that can be activated within an instructional context that

may enhance the learning process (Garris et al., 2002). In other words, outcomes are

affected by the instructional strategies employed in the games, not by the games

themselves (Wolfe, 1997). Leemkuil et al. (2003), too, commented that there is

general consensus that learning with interactive environments such as games,

simulations, and adventures is not effective when no instructional measure or support

is added.

According to Thiagarajan (1998), if not embedded with sound instructional

design, games and simulations often end up as truncated exercises often mislabeled

as simulations. Gredler (1996) further commented that poorly developed exercises

are not effective in achieving the objectives for which simulations are most

appropriate—that of developing students’ problem solving skills. Lee (1999)

64


commented that for instructional prescription we need information dealing with

instructional variables, such as instructional mode, instructional sequence,

knowledge domain, and learner characteristics (Lee, 1999).

Reflection and Debriefing

Instructional strategies that researchers have suggested as beneficial to

learning from games and simulations are reflection and debriefing. Brougere (1999)

argued that a game cannot be designed to directly provide learning; Reflexivity is

required to make transfer and learning possible. Games require reflection, which

enables the shift from play to learning. Therefore, debriefing (or after action review),

which includes reflection, appears to be an essential contribution to research on play

and gaming in education (Brougere, 1999; Leemkuil et al., 2003; Thiagarajan, 1998).

According to Garris et al. (2002), debriefing is the review and analysis of events that

occurred in the game. Debriefing provides a link between what is represented in the

simulation or gaming experience and the real world. It allows the learners to draw

parallels between game events and real-world events. Debriefing allows learners to

transform game events into learning experiences. Debriefing may include a

description of events that occurred in the game, analysis of why they occurred, and

the discussion of mistakes and corrective actions. Garris et al. (2002) argued that

learning by doing must be coupled with the opportunity to reflect and abstract

relevant information for effective learning to occur.

Summary of games and simulations

Computer-based educational games fall into three categories: games,

simulations, and simulation games. While there is debate as to the specific

65


characteristics of each of these three media (e.g., Betz, 1995-1996; Crookall & Arai,

1995; Crookall et al., 1987; Dempsey et al., 2002; Duke, 1995; Garris et al., 2002;

Randel et al., 1992; Ricci et al., 1996), Gredler (1996) provides definitions for the

three media that provides some clear delineations between them. According to

Gredler, games consist of rules, can contain imaginative contexts, are primarily

linear, and include goals as well as competition, either against other players or

against a computer (Gredler, 1996). Simulations display the dynamic relationship

among variables which change over time and reflect authentic causal processes.

Simulations are non-linear and have a goal of discovering causal relationships

through manipulation of independent variables. Simulation games are a blend of

games and simulations (Gredler, 1996). The author of this study does provide one

important modification to Gredler’s definitions.

When Gredler (1996) describes games a linear and simulations as non-linear,

she is referring to their goal structures—games have linear goal structures and

simulations have non-linear goal structures. According to this author, there is another

asepect of games or simulation interaction that can be described as either linear or

non-linear—the intervention structure of the media. Intervention refers to the actions

players or users are allowed to take at any given moment of the game or simulation.

In almost all instances of intervention, both media give at least two choices (e.g., quit

or continue, turn left or turn right, fight or run, increase something or decrease it).

Therefore, for both games and simulations, the intervention structure is non-linear.

While there also debate as to the definition of video games and, more

importantly, whether a computer-based game is a video game, the definitions of

66


computer-based game and video game are beginning to coincide and the two terms

are beginning to be used interchangeably (e.g., Greenfield et al., 1994, 1996;

Kirriemuir, 2002b; Okagaki & Frensch, 1994).

Beginning with the work of Malone (1981), a number of constructs have been

described as providing the motivational aspects of games: fantasy, control and

manipulation, challenge and complexity, curiosity, competition, feedback, and fun.

Fantasy is defined as an environment that evokes “mental images of physical or

social situations that do not exist” (Malone & Lepper, 1987, p. 250). Malone &

Lepper (1987) also commented that fantasies can offer analogies and metaphors, and

Davis and Wiedenbeck (2001) argued that metaphors can help learners feel more

directly involved in a domain.

Control and manipulation promote intrinsic motivation, because learners are

given a sense of control over their choices and actions (deCharms, 1986; Deci,

1975). Challenge embodies the idea that intrinsic motivation occurs when there is a

match between a task and the learner’s skills (Bandura, 1977; Csikszentmihalyi,

1975; Harter, 1978). The task should be neither too hard nor too easy, otherwise, in

both cases, the learner would lose interest (Clark, 1999; Malone & Lepper, 1987).

According to Rieber (1996), curiosity and challenge are intertwined. Curiosity arises

from situations in which there is complexity, incongruity, and discrepancy (Davis &

Wiedenbeck, 2001). Malone and Lepper (1997) argued that curiosity is one of the

primary factors that drive learning.

While Malone (1981) defines competition as important to motivation, studies

on competition with games and simulations have resulted in mixed findings, due to

67


individual learner preferences, as well as the types of reward structures connected to

the competition (e.g., Porter et al. 1990-1991; Yu, 2001). Another motivational

factor in games, feedback, allows learners to quickly evaluate their progress and can

take many forms, such as textual, visual, and aural (Rieber, 1996). Ricci et al. (1996)

argued that feedback can produce significant differences in learner attitudes,

resulting in increased attention to a learning environment. However, Clark (2003)

commented that feedback must be focused on clear learning goals and current

performance results.

The last category contributing to motivation, fun, is possibly an erroneous

category. Little empirical evidence exists for the construct. However, evidence does

support the related constructs of play, engagement, and flow. Play is entertainment

without fear of present of future consequences (Resnick & Sherer, 1994). Webster et

al. (1993) found that labeling software training as serious play improved motivation

and performance. Csikszentmihalyi (1975; 1990) defined flow as an optimal

experience in which a person is so involved in an activity that nothing else seems to

matter. According to Davis and Wiedenbeck (2001), engagement is the feeling of

working directly on the objects of interest in a world, and Garris et al. (2002) argued

that engagement can harness the motivational properties of computer games to

enhance learning and accomplish instructional objectives.

While numerous studies have cited the learning benefits of games and

simulations (e.g., Adams, 1998; Baker et al., 1997; Betz, 1995-1996; Khoo & Koh,

1998), others have found mixed, negative, or null outcomes from games and

simulations, specifically in the relationship of enjoyment of a game to learning from

68


the game (e.g., Brougere, 1999; Dekkers & Donatti, 1981; Druckman, 1995). One of

the problems appears to be non-empirical studies claiming learning outcomes that

cannot be substantiated by the data (see Table 2), as well as empirical studies making

claims not supported by the data (e.g., see Carr & Groves, 1998 in Table 3). Another

problem seems to be the paucity of empirical studies. Of the several thousand articles

on game and simulation studies published in peer reviewed journals in the last 15

years, only 19 were empirical (see Table 3 and O’Neil & Wainess, in preparation).

Another issue in claims attributed to games or simulations is the inaccurate use of

media definition. For examples, the medium used by Ricci et al. (1996) was defined

by the researchers as a game, but the description of the medium met the criteria for a

simulation game. Therefore, any outcomes attributed to the use of games would have

been inaccurate.

Another claim related to the proposed educational benefit of games and

simulations is their motivational characteristics. The assumption is that motivation

always leads to learning. However, a number of researchers suggest that this

relationship may not be true (e.g., Brougere, 1999; Druckman, 1995; Salas, 1998).

Salomon (1984) even contended that a positive attitude can actually indicate less

learning. And Dekkers and Donatti (1981) found that motivation wanes over time, as

the novelty of the game or simulation subsides. While these various arguments

potentially explain the mixed findings with regards to the learning outcomes in

games and simulation research, there is another argument which may provide a better

explanation of the mixed finding.

69


There appears to be consensus among a large number of researcher that the

negative, mixed, or null findings might be related to a lack of sound instructional

design embedded in the games (de Jong & van Joolingen, 1998; Garris et al., 2002;

Gredler, 1996; Lee, 1999; Leemkuil et al., 2003; Thiagarajan, 1998; Wolfe, 1997).

These researchers suggest that it is the instructional design embedded in a medium

and not the medium itself that leads to learning. Instructional design involves the

implementation of various instructional strategies. Among the various strategies,

reflection and debriefing have been cited as critical to learning with games and

simulations. Brougere (1999) argued that games cannot be designed to directly

provide learning; reflection is required to make transfer and learning possible.

Debriefing provides an opportunity for reflection (Brougere, 1999, Garris et al.,

2002; Thiagarajan, 1998).

Assessment of Problem Solving

According to O’Neil’s Problem Solving model (1999), successful problem

solving requires content understanding, problem solving strategies, and self-

regulation. Therefore, proper assessment of problem solving should address all three

constructs.

Measurement of Content Understanding

Davis and Wiedenbeck (2001) commented that meaningful learning results

in an understanding of the basic concepts of the new material through its integration

with existing knowledge. Day et al. (2001), proposed knowledge maps as a method

to measure content understanding. According to Baker and knowledge or concept

mapping is more parsimonious than traditional performance assessment (Baker &

70


Mayer, 1999). In knowledge mapping, “the learner constructs a network consisting

of nodes (e.g., key words or terms) and links (e.g., ‘is part of’, ‘led to’, ‘is an

example of’” (Baker & Mayer, 1999, p. 274). Each node represents a concept in the

domain of knowledge. Each link, which connects two nodes, represents the

relationship between the nodes; that is, the relationship between the two concepts

(Schau & Mattern, 1997). Knowledge structures are based on the premise that people

organize information into patterns that reflect the relationships which exist between

concepts and the features that define them (Day et al., 2001). Day et al. further

commented that, in contrast to declarative knowledge which reflects the amount of

knowledge or facts learned, knowledge structures represent the organization of the

knowledge.

As Schau and Mattern (1997) pointed out, learners should not only be aware

of concepts but also of the connections among them. In a training context,

knowledge structures reflect the degree to which trainees have organized and

comprehended the content of training (Day et al., 2001). Knowledge maps, which are

graphical representations of knowledge structures, have been used as an effective

tool to learn complex subjects (Herl et al., 1996) and to facilitate critical thinking

(West, Pomeroy, Park, Gerstenberger, & Sandoval, 2000). Several studies also

revealed that knowledge maps are not only useful for learning, but are a reliable and

efficient measurement of content understanding (Herl et al., 1999; Ruiz-Primo,

Schultz, & Shavelson, 1997). The results of a study by Day et al. (2001) indicated

that knowledge structures are predictive of both skill retention and skill transfer and

can therefore be a viable indices of training outcomes. Ruiz-Primo et al. (1997)

71


proposed a framework for conceptualizing knowledge maps as a potential

assessment tool in science, because they allow for organization and discrimination

between concepts.

Ruiz-Primo et al. (1997) stated that, as an assessment tool, knowledge maps

are identified as a combination of three components: (a) a task that allows a student

to exhibit his or her content understanding in the specific domain (b) a format for the

student’s responses, and (c) a scoring system by which the student’s knowledge map

could be accurately evaluated. Chuang (2003) modified this framework to serve as

an assessment specification using a concept map. Researchers have successfully

applied knowledge maps to measure students’ content understanding in science for

both high school students and adults (e.g., Chuang, 2003; Herl et al., 1999; Schacter

et al., 1999; Schau et al., 2001). For example, Schau et al. (2001) used select-and-

fill-in knowledge maps to measure secondary and postsecondary students’ content

understanding of science in two studies. The results of the participant’s performance

on the knowledge maps correlated significantly with that of a multiple choice test, a

traditional measure of learning (r= .77 for eighth grade and r=. 74 for seventh grade),

providing validity to the use of knowledge maps to assess learning outcomes.

CRESST developed a computer-based knowledge mapping system, which

measures the deeper understanding of individual students and teams, and reflects

thinking processes in real-time (Chung et al., 1999; O’Neil, 1999; Schacter et al.,

1999). The computer-based knowledge map has been used successfully in a number

of studies (e.g., Chuang, 2003; Chung et al., 1999; Hsieh, 2001; Schacter et al.,

1999).

72


In the four studies, the map contained 18 concepts of environmental science,

and seven links for relationships, such as “cause,” “influence,” and “used for.”

Subjects were asked to create a knowledge map in the computer-based environment.

In the study conducted by Schacter et al. (1999) students were evaluated by creating

individual knowledge maps, after searching a simulated World Wide Web

environment. In studies conducted by Chung et al. (1999), Hsieh (2001), and Chuang

(2003), two students constructed a group map cooperatively through networked

computers. Results of the cooperative studies showed that using networked

computers to measure group processes was feasible. Figures 2 and 3 shows a screen

shot of the knowledge mapping software used for this study, which is similar to the

knowledge map software used for the three studies discussed above.

Figure 2: Knowledge Map User Interface Displaying 3 Concepts and 2 Links

As seen in Figure 2, the computer screen was divided into three major

sections. The bottom section was for selecting the interaction mode. The middle

73


section is where one of the team members constructed the knowledge map. The top

section contained four menu items: “Session,” “Add Concept,” “Available Links,”

and “About.” Figure 3 shows the drop-down menu that appeared when “Add

Concept” was clicked. Clicking when the mouse pointer was over a concept added

that concept to the knowledge map. Figure 2 shows three concepts that were added:

desk, safe, and key. Figure 2 also shows links that were added to the concept map by

(A) clicking on one concept on the screen, (B) holding the mouse button down,

dragging to another concept, and letting go, which opened a ‘link’ dialog box, then

(C) selecting an appropriate link from a pull-down menu on the dialog box, and (D)

clicking the OK button on the dialog box to close the dialog box and complete the

link process.

Figure 3: Adding Concepts to the Knowledge Map

74


Measurement of Problem Solving Strategies

According to Baker and Mayer (1999), “Problem solving is cognitive


obvious method of solution is available to the problem solver” (p. 272). Simply put,

problem solving is mental activity aimed at finding a solution to a problem. Problem

solving strategies, which are almost always procedural, can be categorized as

domain-independent (-general) and domain-dependent (-specific; Alexander, 1992;

Bruning, Schraw, & Ronning, 1999; O’Neil, 1999; Perkins & Salomon, 1989).

Domain-specific problem solving knowledge is knowledge about a particular field of

study or a subject, such as the application of equations in a math question, the

application of a formula in a chemistry problem, or the specific strategies to be

successful in a game. Domain-general problem knowledge is the broad array of

problem solving knowledge that is not linked with a specific domain, such as the

application of multiple representations and analogies in a problem solving task or the

use of Boolean search strategies in a search task (Chuang, 2003).

Transfer questions have been examined as an alternative to performing

transfer tasks. For example, in a recent study which involved computer-based

delivery of information on how lightning worked, to examine the split-attention

effect in multimedia learning, Mayer and Moreno (1998) assessed participants’

problem solving strategies through a list of transfer questions. There were four

transfer questions: “What could you do to decrease the intensity of lightning?”

“Suppose you see clouds in the sky, but no lightning. Why not?” “What does air

temperature have to do with lightning?” and “What causes lightning?” The

75


researchers had generated a list of 12 acceptable responses to the four questions and

participants received points for matching those responses. Participant responses to

the transfer questions were positively correlated with performance, indicating that

transfer questions are a viable alternative to more traditional methods of measuring

retention and transfer, such as tests and novel problem solving (Mayer & Moreno,

1998).

Measurement of Self-Regulation

While Brunning et al. (1999) commented that some researchers believe self-

regulation includes three core components—metacognitive awareness, strategy use,

and motivational control, according to the O’Neil Problem Solving model (O’Neil,

1999), self-regulation is composed of only two core components: metacognition and

motivation. Strategy use is a separate construct that encompasses domain-specific

and domain-general knowledge. Within the O’Neil (1999) model, metacognition is

comprised of two subcategories, planning and self-monitoring (Hong & O’Neil,

2001; O’Neil & Herl, 1998; Pintrich & DeGroot, 1990) and motivation encompasses

mental effort and self-efficacy (Zimmerman, 1994, 2000).

O’Neil and Herl (1998) developed a trait self-regulation questionnaire

examining the four components of self-regulation (planning, self-monitoring, mental

effort, and self-efficacy). As defined by the O’Neil Problem Solving model (O’Neil,

1999), of the four components, planning is the first step in problem solving, since

learners must have a plan to achieve the proposed goal. Self-efficacy is one’s belief

in his or her capability to accomplish a task (Davis & Wiedenbeck, 2001), and

mental effort is amount of mental effort exerted on a task. Self-monitoring occurs

76


throughout problem solving and involves comparing one’s current state to the goal

state, to determine if the current strategy is effective or whether modifications should

be made.

In the trait self-regulation questionnaire developed by O’Neil and Herl

(1998), planning, self-monitoring, self-efficacy, and effort are assessed using eight

questions each, for a total of thirty-two questions. The reliability of this self-

regulation inventory has been established in previous studies. For example, in the

research conducted by Hong and O’Neil (2001), the reliability estimates (coefficient

alpha) of the four subscales of self-regulation—planning, self-checking, mental

effort, and self-efficacy—were .76, .86, .83, and .85 respectively; The research has

also provided evidence for construct validity.

Summary of Problem Solving Assessment

Problem solving is cognitive processing directed at achieving a goal when no

solution method is obvious to the problem solver (Baker & Mayer, 1999). In the

O’Neil Problem Solving model (O’Neil, 1999: see Figure 1 in this dissertation),

problem solving is comprised of three components: content understanding, problem

solving strategies, and self-regulation. Content understanding refers to domain

knowledge. Problem solving strategies can be categorized into two types: domain-

independent (-general) and domain-dependent (-specific) problem solving strategies.

Self-regulation includes two sub-categories: metacognition and motivation.

Metacognition is composed of self-monitoring and planning. Motivation is

comprised of effort and self-efficacy.

77


Knowledge maps have been shown to be a reliable and efficient method for

the measurement of content understanding. CRESST has developed a simulated

World Wide Web space that incorporates knowledge mapping software to evaluate

problem solving strategies such as information searching strategies and feedback

inquiring strategies. Research has shown that computer-based problem solving

assessments are economical, efficient and valid measures that employ contextualized

problems that require students to think for extended periods of time and to indicate

the problem solving heuristics they were using and why.

Problem solving strategies are almost always procedural (Alexander, 1992;

Bruning et al., 1999; O’Neil, 1999, Perkins & Salomon, 1989). Domain-specific

problem solving strategies are applied to a particular field of study or subject, such as

the application of an equation to solve a math question. Domain-general problem

solving strategies refer to a broad array of problem solving knowledge not linked to a

specific domain, such as the application of multiple representations and analogies in

a problem solving task or the use of Boolean search strategies in a search task

(Chuang, 2003). Problem solving strategy transfer questions have been shown to be

an effective alternative to performing problem solving transfer tasks (Mayer &

Moreno, 1998).

According to the O’Neil Problem Solving Model (O’Neil, 1999), self-

regulation is composed of two core components: metacognition and motivation.

Metacognition is further analyzed into planning and self-monitoring (Hong &

O’Neil, 2001; O’Neil & Herl, 1998; Pintrich & DeGroot, 1990). Motivation is

further analyzed into mental effort and self-efficacy (Zimmerman, 1994, 2000).

78


Planning is the first step in problem solving (O’Neil, 1999). Self-efficacy is one’s

belief in the capacity to achieve a proposed goal (Davis & Wiedenbeck, 2001).

Mental effort is the amount of mental effort exerted on a task (Davis & Wiedenbeck,

2001). Self-monitoring occurs throughout the problem solving process and involves

comparing one’s current state to a goal state, to determine if the current strategy is

effective or whether modifications should be made (O’Neil, 1999). O’Neil and Herl

(1998) developed a trait self-regulation questionnaire to examine the four self-

regulation components (planning, self-monitoring, mental effort, and self-efficacy).

The questionnaire is comprised of 32 questions; eight questions for each of the four

self-regulation components. Research conducted by Hong and O’Neil (2001) has

shown reliability estimates for the planning, self-checking, mental effort, and self-

efficacy portions of the questionnaire of .76, .86, .83, and .85 respectively. The

research also provided evidence for construct validity.

Scaffolding

As discussed earlier, cognitive load theory (Paas et al., 2003) is concerned

with methods for reducing the amount of cognitive load placed on working memory

during learning and problem solving activities. Clark (2003b) commented that

instructional methods must also keep the cognitive load from instructional

presentations to a minimum. Scaffolding is considered a viable instructional method

that assists in cognitive load reduction. There are a number of definitions of

scaffolding in the literature. Chalmers (2003) defines scaffolding as the process of

forming and building upon a schema (Chalmers, 2003). In a related definition, van

Merrionboer et al. (2003) defined the original meaning of scaffolding as all devices

79


or strategies that support students’ learning. More recently, van Merrienboer, Clark,

and de Croock (2002) defined scaffolding as the process of diminishing (fading)

support as learners acquire more expertise. Allen (1997) defined scaffolding as the

process of training a student on core concepts and then gradually expanding the

training. To summarize, the four definitions of scaffolding involve the development

of simple to complex schema, all devices that support learning, the process of

diminishing (fading) support during learning, and the process of building learning

from basic concepts to complex knowledge, respectively. Ultimately, the core

principle embodied in each of these definitions is that scaffolding is concerned with

controlling the amount of cognitive load imposed by learning, and each reflects a

philosophy or approach to controlling or reducing that load. For the purposes of this

review, all four definitions of scaffolding will be considered.

As defined by Clark (2001), instructional methods are external

representations of internal cognitive processes that are necessary for learning but

which learners cannot or will not provide for themselves. They provide learning

goals (e.g., demonstrations, simulations, and analogies: Alessi, 2000; Clark 2001),

monitoring (e.g., practice exercises: Clark, 2001), feedback (Alessi, 2000; Clark

2001; Leemkuil et al., 2003), and selection (e.g., highlighting information: Alessi,

2000; Clark, 2001). Alessi (2000) added that instructional methods include: giving

hints and prompts before student actions; providing coaching, advice, or help

systems; and providing dictionaries and glossaries. Jones et al. (1995) added

advance organizers, graphical representations of problems, and hierarchical

80


knowledge structures to the list of instructional methods. Each of these examples is a

form of scaffolding.

In learning by doing in a virtual environment, students can actively work in

realistic situations that simulate authentic tasks for a particular domain (Mayer et al.,

2002). A major instructional issue in learning by doing within simulated

environments concerns the proper type of guidance (i.e. scaffolding), that is, how

best to create cognitive apprenticeship (Mayer et al. 2002). Mayer and colleagues

(2002) also commented that their research shows that discovery-based learning

environments can be converted into productive venues for learning when appropriate

cognitive scaffolding is provided; specifically, when the nature of the scaffolding is

aligned with the nature of the task, such as pictorial scaffolding for pictorially-based

tasks and textual-based scaffolding for textually-based tasks. For example, in a

recent study, Mayer et al. (2002) found that students learned better from a computer-

based geology simulation when they were given some graphical support about how

to visualize geological features, as opposed to textual or auditory guidance.

Graphical Scaffolding

According to Allen (1997), selection of appropriate text and graphics can aid

the development of mental models, and Jones et al. (1995) commented that visual

cues such as maps and menus as advance organizers help learners conceptualize the

organization of the information in a program. A number of researchers support the

use of maps as visual aids and organizers (Benbasat & Todd, 1993: Chou & Lin,

1998; Chou et al., 2000; Farrell & Moore, 2000-2001; Ruddle et al, 1999)

81


Chalmers (2003) defined graphic organizers as organizers of information in a

graphic format, which act as spatial displays of information that can also act as study

aids. Jones et al. (1995) argued that interactive designers should provide users with

visual or verbal cues to help them navigate through unfamiliar territory. Overviews,

menus, icons, or other interface design elements within the program should serve as

advance organizers for information contained in the interactive program (Jones et al.,

1995). In addition, the existence of virtual bookmarks enables recovery from the

possibility of disorientation; loss of place (Dias, Gomes, & Correia, 1999). However,

providing such support devices does not guarantee learners will use them. For

example, in an experiment involving a virtual maze, Cutmore et al. (2000) found

that, while landmarks provided useful cues, males utilized them significantly more

often than females did.

Navigation maps

Cutmore et al. (2000) define navigation as “…a process of tracking one’s

position in a physical environment to arrive at a desired destination” (p. 224). A

route through the environment consists of either a series of locations or continuous

movement along a path. Cutmore et al. further commented that “Navigation becomes

problematic when the whole path cannot be viewed at once but is largely occluded

by objects in the environment’” (p. 224). The occluding objects may include internal

walls or large environmental features such as trees, hills, or buildings. Under these

conditions, one cannot simply plot a direct visual course from the start to finish

locations. Rather, knowledge of the layout of the space is required. Navigation maps

or other descriptive information may provide that knowledge (Cutmore et al., 2000).

82


Effective navigation of a familiar environment depends upon a number of

cognitive factors. These include working memory for recent information, attention to

important cues for location, bearing and motion, and finally, a cognitive

representation of the environment which becomes part of a long-term memory, a

cognitive map (Cutmore et al., 2000). According to Yair et al. (2001), the loss of

orientation and “vertigo” feeling which often accompanies learning in a virtual-

environment is minimized by the display of a traditional, two-dimensional map. The

map helps to navigate and to orient the user, and facilitates an easier learning

experience. Dempsey et al. (2002) also commented that an overview of player

position was considered an important feature in adventure games.

A number of experiments have examined the use of navigation maps in

virtual environments. Chou and Lin (1998) and Chou et al. (2000) examined various

navigation map types, with some navigation maps offering global views of the

environment (global navigation map) and others offering more localized views (local

navigation map), based on the learner’s current location. One hundred twenty one

college students participated in the Chou and Lin (1998) study. Five groups were

created, based on four navigation map variations; a global map of the entire 94 node

hierarchical knowledge structure (the entire hypermedia environment), a series of

local maps for each knowledge area of the environment, a tracking map that updated

according the participant’s location with the participant’s location always in the

center and showing one level of nodes above and two below the current position, and

a no-map situation. One group was assigned to the global map, one to the local map,

one to the tracking map, and one two no map. A fifth group had access to all three

83


map types (global, local, and tracking). After being given instruction on their

respective navigation tools and time to practice, subjects were given 10 search tasks

and an addition 30 minutes to browse the hypermedia environment, after which they

answered posttest questions and an attitude questionnaire. Subjects also created a

knowledge map.

Results of the Chou and Lin (1998) study indicated that the search efficiency

(search speed) for the all map and global map groups were significantly lower than

for the other three groups (local map, tracking map, and no map), indicating benefits

from using the global map or all maps (which included the global map). There was

not significance between the all map and global map groups. Knowledge map

creation for the all map and global map groups were also significantly higher than

for the tracking map group, but not the local map or no map groups. Overall, the

results of the Chou and Lin (1998) study suggest that use of a global map or use of a

combination of maps, including the global map, results in greater search efficiency

and greater content understanding (as indicated by knowledge map development)

then either local maps or no map. Additionally, there were no differences found

between the use of a local map versus no map, suggesting no cognitive value to a

local map. With regard to attitude, there were no difference by map type for any of

the attitudinal scales, including attitude toward the learning experience, usability of

the system, and disorientation.

As with Chou and Lin (1998) study, the Chou et al. (2000) study, which

involved over one hundred college students, showed that the type of navigation map

used affected performance. However, some findings were in contrast to the earlier

84


study. Those who used a global navigation map performed significantly better than

both the local map and no map groups with regard to the number of areas visited in

order to accomplish the search task. There was no difference in performance between

the local map and the no map groups. The number of times areas were revisited was

also significantly lower for the global map group than for either the no map or the

local map groups, but there was no difference between the no map and local map

groups. In the third measure, development of a knowledge map, the no map group’s

performance was significantly higher than the local map’s performance. The global

map group’s score was slightly lower than then no map group’s score and fell just

short of significance over the local map’s score. This differed from the earlier from

the earlier study which found a significant difference between the global map over

no map, suggesting that map use might not be a primary factor in developing content

understanding.

Results of the Chou et al (2000) study indicated that map type can affect

performance in a search task. A global map resulted in better performance than a

local map or no map with regards to navigation (search speed and revisiting sites),

while performance on knowledge map creation by the no map group was

significantly better than for those who use a local map and only slightly better than

for those who used a global map. In other words, accomplishment of a problem

solving task was best with a global map while understanding of a problem solving

task or environment was best with either no map or with a global map. Results of

the two Chou and colleague studies (Chou & Lin 1998 & Chou et al., 2000) suggest

that global map use can improve search speed and reduce revisiting locations. The

85


mixed results of the two studies suggest that map use may not influence content

understanding.

According to Tkacz (1998), soldiers use navigation maps as tools, which

involve spatial reasoning, complex decision making, symbol interpretation, and

spatial problem solving. In her study involving 105 marines, Tkacz (1998) examined

the procedural components of cognitive maps required for using and understanding

topographic navigation maps, stating that navigation map interpretation involves

both top-down (retrieved from long-term memory) and bottom-up (retrieved from

the environment and the navigation map) procedures. Therefore, Tkacz examined the

cognitive components underlying navigation map interpretation to assess the

influence of individual differences on course success and on real world position

location. In addition, Tkacz, related position location ability to video game

performance in a simulated environment. Performance measures consisted of real-

world position location (in the field), a map reading readiness exercise (using a map),

and simulated travel in a videogame environment (a 3-dimensional maze, with

movement in six directions; North, South, East, West, Up, and Down). The goal of

the maze was to move as quickly as possible through the 125 room structure to a

goal room and then find the exit door.

All participants completed a map reading pretest to assess basic map skills.

The treatment group then received 15 hours of geographical training covering six

topics: terrain association, contour intervals, elevation, landforms, slope type, and

slope steepness. After the training, all groups completed spatial tests and a

geography test. The geography test assessed the six skills covered in geographical

86


training the treatment group received. Additional participant data obtained from

armed services vocational aptitude tests administered to each participant during

military enlistment were also utilized.

According to Tkacz, the geographical instruction significantly improved the

ability to perform terrain association and relate the real world scenes to

topographical map representations. Results of the study also indicated that

orientation and, to a lesser extent, reasoning ability are important for map

interpretation. Video game performance was affected by all spatial skills, and

particularly orientation and mental rotation (visualization), with high ability subjects

escaping the maze faster than lower ability subjects. Video game performance was

also affected by map reading ability, with better performance by those demonstrating

better map reading performance. It should be noted that, while Tkacz referred to the

maze as a game, it appears to fit the Gredler’s (1996) definition of a simulation

game, not a game.

Mayer et al. (2002) commented that a major instructional issue in learning by

doing within simulated environments concerns the proper type of guidance, which

they refer to as cognitive apprenticeship. The investigators used a geological gaming

simulation, the Profile Game, to test various types of guidance structures (i.e.,

strategy modeling), ranging from no guidance to illustrations (i.e., pictorial aids) to

verbal descriptions to pictorial and verbal aids combined. The Profile Game is based

on the premise, “Suppose you were visiting a planet and you wanted to determine

which geological feature is present on a certain portion of the planet’s surface”

(Mayer et al., p. 171). While exploring, you cannot directly see the features, so you

87


must interpret data indirectly, through probing procedures. The experimenters

focused on the amount and type of guidance needed within the highly spatial

simulation.

Though a series of experiments, Mayer et al. (2002) found that pictorial

scaffolding, as opposed the verbal scaffolding, is needed to enhance performance in

a visual-spatial task. In the final experiments of the series, participants were divided

into verbal scaffolding, pictorial scaffolding, both, and no scaffolding groups.

Participants who received pictorial scaffolding solved significantly more problems

than did those who did not receive pictorial scaffolding. Students who received

strategic scaffolding did not solve significantly more problems than students who did

not receive strategic scaffolding. While high-spatial participants performed

significantly better than low-spatial students, adding pictorial scaffolding to the

learning materials helped both low- and high-spatial students learn to use the Profile

Game. Students in the pictorial-scaffolding group correctly solved more transfer

problems than students in the control group. However, pictorial scaffolding did not

significantly affect the solution time (speed) of either low- or high-spatial

participants. Overall, adding pictorial scaffolding to the learning materials lead to

improved performance on a transfer task for both high- and low-spatial students in

the Profile Game (Mayer et al., 2002).

Contiguity effect

The contiguity effect addresses the cognitive load imposed when multiple

sources of information are separated (Mayer & Moreno, 2003; Mayer & Sims, 1994;

Mayer et al., 1999; Moreno & Mayer, 1999). There are two forms of the contiguity

88


effect: spatial contiguity and temporal contiguity. Temporal contiguity occurs when

one piece of information is presented prior to other pieces of information (Mayer &

Moreno, 2003; Mayer et al., 1999; Moreno & Mayer, 1999). Spatial contiguity

occurs when modalities are physically separated (Mayer & Moreno, 2003). This

study is concerned with spatial contiguity, since the printed navigation maps will be

spatially separated from the 3-D video game environment. Contiguity results in split-

attention (Moreno & Mayer, 1999).

Split Attention Effect

When dealing with two or more related sources of information (e.g., text and

diagrams), it’s often necessary to integrate mentally corresponding representations

(e.g., verbal and pictorial) to construct a relevant schema to achieve understanding.

When different sources of information are separated in space or time, this process of

integration may place an unnecessary strain on limited working memory resources,

resulting in impairment in learning (Atkinson et al., 2000; Mayer & Moreno, 1998;

Tarmizi & Sweller, 1988). Mayer (2001) commented that the split attention effect

can be resolved by placing the components near each other; for example, placing text

labels near their related imagery in an illustration. In this study, the printed

navigation maps are spatially separated from the 3-D video game environment,

thereby inducing the split-attention effect.

Summary of scaffolding

Depending upon the researcher, scaffolding has several meanings: the

process of forming and building upon a schema (Chalmers, 2003); all devices or

strategies that support learning (van Merrionboer et al., 2003), the process of

89


diminishing support as learners acquire expertise (van Merrionboer et al., 2002); and

the process of training a student on core concepts and then gradually expanding the

training. What these four definitions have in common is that scaffolding is related to

providing support during learning, to control or limit cognitive load.

Clark (2001) described instructional methods as external representations the

internal metacognitive processes of selecting, organizing, and integrating.

Instructional methods also provide learning goals (Alessi, 2000; Clark, 2001),

monitoring (Clark, 2001), feedback (Alessi, 2000; Clark, 2001; Leemkuil et al.,

2003), selection (Alessi, 2000; Clark, 2001), hints and prompts (Alessi, 2000), and

various advance organizers (Jones et al., 1995). Each of these components either

reflects a form of scaffolding or reflects a need for scaffolding.

Mayer et al (2002) argued that a major instructional issue in learning by

doing within simulated environments concerns the proper type of guidance (i.e.,

scaffolding). One form of scaffolding is graphical scaffolding. According to Allen

(1997), selection of appropriate text and graphics can aid the development of mental

models, and Jones et al. (1995) commented that visual cues such as maps help

learners conceptualize the organization of the information in a program (i.e., the

learning space). A number of studies have supported the use of maps as visual aids

and organizers (Benbasat & Todd, 1993: Chou & Lin, 1998; Ruddle et al, 1999,

Chou et al., 2000; Farrell & Moore, 2000-2001)

According to Allen (1997), selecting of appropriate text and graphics can aid

the development of mental models. Jones et al. (1995) commented that visual cues

such as maps and menus as advance organizers help learners conceptualize the

90


organization of information. Graphic organizers arrange information in a graphic

format, which act as spatial displays of information that can also act as study aids

(Chalmers, 2003).

Cobb argued that cognitive load can be distributed to external media (Cobb,

1997). One type of external media is a navigation map. When navigating occluded

environments, where obstructions prevent viewing of or knowledge of an entire path,

navigation maps can provide that knowledge (Cutmore et al, 2000). According to

Yair et al. (2001), the disorientation that accompanies learning in a virtual

environment can be minimized by use of a traditional, two-dimensional map.

A number of experiments have examined the use of navigation maps in

virtual environment. Chou and Lin (1998) examined the use of various map types to

navigation during a search and information gathering task in a web-like environment.

Five map variations were examined: global map, two local map types, no map, and

all maps. Those using the global map or all maps performed searches more

efficiently (faster and with less revisiting) than those using the local maps or no

maps. Results of knowledge map creation was mixed, with the global and all map

groups performing better than one local map type but not the other local map type or

the no map group, suggesting that map type may not affect content understanding.

Based on results of the 1998 study, the Chou et al (2000) study examined

map use with three map types: global map, local map, and no map. Similar to the

first study, the global map group performed significantly higher than the local and no

map groups. Also similar to the first study, the number of revisits to web pages was

significantly lower for the global map group as compared to the local and no map

91


groups. In contrast to the first study, knowledge map creation by the no map group

performed significantly higher than either the local map group. The global map

group’s performance was equivalent to the no map group, but fell short of being

significantly better than the local map group. Results of the two Chou and colleague

studies (Chou & Lin 1998 & Chou et al., 2000) suggest that global map use can

improve search speed and reduce revisiting locations. The mixed results of the two

studies suggest that map use may not influence content understanding.

Tkacz (1998) stated that navigation map interpretation involves both top-

down (retrieved from long-term memory) and bottom-up (retrieved from the

environment and the navigation map) procedures. Tkacz (1998) examined the

cognitive components underlying navigation map interpretation that assess the

influence of individual differences. Tkacz also related position location ability to

video game performance in a simulated environment (a maze). Results of the Tkacz

(1998) study indicated that orientation, and to some extent reasoning ability, were

important for map interpretation. Video game performance was affected by all

spatial skills, and particularly by orientation and mental rotation (visualization).

Video game performance was also affected by map reading ability. While Tkacz

referred to the maze as a game, it appears to fit Gredler’s (1996) definition of a

simulation game, not a game.

Using a geological simulation game, the Profile Game, Mayer et al. (2002)

examined various types of guidance structures, ranging from no guidance to

illustrations (i.e., pictorial aids) to verbal descriptions to pictorial and verbal aids

combined. In the Profile Game, participants needed to determine surface features of

92


an environment, without directly observing the features. Results of the experiment

indicated that the type of scaffolding provided should be aligned with the type of

task. In other words, graphical scaffolding should be provided during graphical tasks,

auditory scaffolding for auditory tasks, and textual scaffolding for textual tasks

(Mayer et al., 2002).

While graphical scaffolding appears to be beneficial, there are potential

problems associated with this type of scaffolding. One such problem is referred to as

the contiguity effect, which refers to the cognitive load imposed with multiple

sources of information are separated (Mayer & Moreno, 2003; Mayer et al., 1999).

There are two forms of the contiguity effect: spatial contiguity and temporal

contiguity. Temporal contiguity occurs when one piece of information is presented

prior to other pieces of information Spatial contiguity occurs when information is

physically separated (Mayer & Moreno, 2003). This study potentially imposes

spatial contiguity, since the navigation map is presented on a piece of paper which,

depending on where the participant places the map, is separated from the computer

screen. The contiguity effect results in split attention (Moreno & Mayer, 1999).

According to the split attention effect, when information is separated by space

of time, the process of integrating the information may place an unnecessary strain

on limited working memory resources (Atkinson et al., 2000; Tarmizi & Sweller,

1998, Mayer, 2001). Placing the information next to each other can reduce the effect

(Mayer, 2001).

Summary of the Literature Review

93


Cognitive Load Theory is based on the assumptions of a limited working

memory with separate channels for auditory and visual/spatial stimuli, and a virtually

unlimited capacity long-term memory that stores schemas of varying complexity and

levels of automation (Brunken et al., 2003). According to Paas et al. (2003),

cognitive load refers to the amount of load placed on working memory. Cognitive

load can be reduced through effective use of the auditory and visual/spatial channels,

as well as schemas stored in long-term memory. There are three types of cognitive

load that can be described in relation to a learning or problem solving task: intrinsic

cognitive load (load from the actual mental processes involved in creating schema),

germane cognitive load (load from the instructional processes that deliver the to-be-

learned content), and extraneous cognitive load (all other load). An important goal of

instructional design is to balance intrinsic, germane, and extraneous cognitive loads

to support learning outcomes (Brunen et al., 2003).

Working memory refers to the limited capacity of holding and processing

chunks of information. Miller (1956) defines that limitation as five to nine chunks.

Paas et al. (2003) added that limitations might be greater when dealing with novel

information, with working memory only able to handle as few as two or three novel

chunks of information.

The three components of working memory (the central executive, the

visuospatial sketchpad, and the phonological loop) are limited in capacity and

temporary (Baddeley, 1986; Brunken et al., 2003). By contrast, long term memory,

which stores information as schema, is permanent and has an unlimited capacity

(Tennyson & Breuer, 2002). Schemas, which are cognitive constructs that

94


incorporate multiple elements of information into a single element (Paas et al.,

2003), reduce working memory load by treating those multiple elements as one

chunck of information. Through practice, schemas can also operate under automatic,

rather than controlled, processing, requiring minimal working memory resources

(Clark, 1999; Kalyuga et al., 2003; Mousavi et al., 1995). Because of their cognitive

benefits, the primary goals of instruction are construction (chunking) of and

automation of schemas (Paas et al., 2003).

Elaboration and reflection are processes involved in the development of

schemas and mental models. Elaboration consists of creating of a semantic event

(Kees & Davies, 1990) and reflection encourages learners to examine information

and processes (Atkinson et al., 2003). According to Allen (1997), mental models,

which are internal representations of our understanding of external processes, differ

from schema which can model other types of knowledge, not just processes.

Metacognition (also know as the central executive), is the management of

cognitive processes (Jones et al., 1995), as well as the awareness of ones own mental

processes (Anderson et al., 2001). The metacognitive components appearing in most

cognitive models are selecting (attending to relevant information), organizing

(building connections between pieces of information), and integrating (connecting

new information to prior knowledge; Harp & Mayer, 1998).

Meaningful learning is defined as deep understanding of the material and is

reflected in the ability to apply what was taught to new situations; i.e., transfer or

problem solving transfer (Mayer & Moreno, 2003). Meaningful learning requires

effective metacognitive skills (Jones et al., 1995). Related to meaningful learning is

95


mental effort, which refers to the cognitive capacity allocated to a task. Mental effort

is affected by motivation, which in turn cannot exist without goals (Clark, 2003d).

Goals are further affected by self-efficacy; the belief in one’s ability to successfully

carry out a particular behavior (Davis & Wiedenbeck, 2001).

While mental effort is affected by motivation, one does not necessarily lead

to the other (Clark, 2003d; Salomon, 1983). A number of factors can affect

motivation, including prior knowledge, strategy knowledge, personal-motivation

states (e.g., self-efficacy and intrinsic motivation) and knowledge of oneself (e.g.,

goals and self-perceptions; Browkowski et al., 1990). The difficulty of a task also

affects motivation. Tasks that are too easy or too hard tend to reduce motivation

(Clark, 1999). Expectancy-value theory proposes that the probability of behavior

depends on the value of a goal and the expectancy of attaining the goal (Coffin &

MacIntyre, 1999). Task value is affected by a number of factors including the

intrinsic value of a goal and its attainment value (Corno & Mandinah, 1983).

Related to meaningful learning is problem solving, which is “cognitive


obvious methods of solution is available to the problem solver” (Baker & Mayer,

1999, p. 272). O’Neil’s Problem Solving model (O’Neil, 1999) defines three core

constructs of problem solving: content understanding, problem solving strategies,

and self-regulation. Most of these components is further defined by subcomponents.

Content understanding refers to domain knowledge. Problem-solving strategies refer

to both domain-specific and domain-independent strategies. Self-regulation is

96


comprised of metacognition (planning and self-monitoring) and motivation (mental

effort and self-efficacy; O’Neil, 1999, 2002).

Learner control, which is inherent in interactive computer-based media,

allows for control of pacing and sequencing (Barab et al., 1999). It also can induce

cognitive overload in the form of disorientation—loss of place (Chalmers, 2003)—

and is a potential source for extraneous cognitive load. These issues may be a cause

of mixed reviews of learner control (Bernard, et al, 2003; Niemiec et al., 1996;

Steinberg, 1989), particularly in relationship to novices versus experts (Clark,

2003c).

Computer-based educational games fall into three categories: games,

simulations, and simulation games. Games consist of rules, can contain imaginative

contexts, are primarily linear, and include goals as well as competition, either against

other players or against a computer (Gredler, 1996). Simulations display the dynamic

relationship among variables which change over time and reflect authentic causal

processes. Simulations have a goal of discovering causal relationships through

manipulation of independent variables. Simulation games are any blend of games

and simulations (Gredler, 1996). The terms computer-based game and video game

are used interchangeably (Kirriemuir, 2002b). While games have been described as

linear and simulations as non-linear, this refers to the goal structures of the media. In

terms of intervention structure, both media are non-linear. In other words, at each

intervention point the user or participant can select from either two choices (e.g., quit

or continue, increase or decrease something, go left or go right).

97


Beginning with the work of Malone (1981), a number of constructs have been

described as providing the motivational aspects of games: fantasy, control and

manipulation, challenge and complexity, curiosity, competition, feedback, and fun.

Fantasy evokes “mental images of physical or social situations that do not exist”

(Malone & Lepper, 1987, p. 250). Control and manipulation promote intrinsic

motivation, because learners are given a sense of control over their choices and

actions (deCharms, 1986; Deci, 1975). Challenge embodies the idea that intrinsic

motivation occurs when there is a match between a task and the learner’s skills

(Bandura, 1977; Csikszentmihalyi, 1975; Harter, 1978). For challenge to be

effective, the task should be neither too hard nor too easy, otherwise the learner will

lose interest (Clark, 1999; Malone & Lepper, 1987). Curiosity is related to challenge

and arises from situations in which there is complexity, incongruity, and discrepancy

(Davis & Wiedenbeck, 2001).

Studies on competition with games and simulations have resulted in mixed

findings, due to individual learner preferences, as well as the types of reward

structures connected to the competition (see, for example, Porter et al., 1990-1991;

Yu, 2001). Another motivational factor in games, feedback, allows learners to

quickly evaluate their progress and can take many forms, such as textual, visual, and

aural (Rieber, 1996). Ricci et al. (1996) argued that feedback can produce significant

differences in learner attitudes, resulting in increased attention to a learning

environment. However, Clark (2003) commented that feedback must be focused on

clear performance goals and current performance.

98


The last category contributing to motivation, fun, is possibly an erroneous

category. Little empirical evidence exists for the construct. However, evidence does

support the related constructs of play, engagement, and flow. Play is entertainment

without fear of present of future consequences (Resnick & Sherer, 1994).

Csikszentmihalyi (1975, 1990) defines flow as an optimal experience in which a

person is so involved in an activity that nothing else seems to matter. According to

Davis and Wiedenbeck (2001), engagement is the feeling of working directly on the

objects of interest in a world, and Garris et al. (2002) argued that engagement can

help to enhance learning and accomplish instructional objectives.

While numerous studies have cited the learning benefits of games and

simulations (e.g., Adams, 1998; Baker et al., 1997; Betz, 1995-1996; Khoo & Koh,

1998), others have found mixed, negative, or null outcomes from games and

simulations, specifically in the relationship of enjoyment of a game to learning from

the game (e.g., Brougere, 1999; Dekkers & Donatti, 1981; Druckman, 1995). Part of

the problem comes from unsupported claims from non-empirical studies (see Table

2) and even from empirical studies (e.g., Carr and Groves, 1998). However, there

appears to be consensus among a large number of researchers with regards to the

negative, mixed, or null findings, suggesting that the cause might be a lack of sound

instructional design embedded in the games (de Jong & van Joolingen, 1998; Garris

et al., 2002; Gredler, 1996; Lee, 1999; Leemkuil et al., 2003; Thiagarajan, 1998;

Wolfe, 1997). Among the various instructional strategies, reflection and debriefing

have been cited as critical to learning with games and simulations.

99


An important component in research on the effectiveness of educational

games and simulations is the measurement and assessment of performance outcomes

from the various instructional strategies embedded into the games or simulations,

such as problem solving tasks. Problem solving is cognitive processing directed at

achieving a goal when no solution method is obvious to the problem solver (Baker &

Mayer, 1999). The O’Neil Problem Solving model (O’Neil, 1999) includes three

components: content understanding; solving strategies—domain-independent (-

general) and domain-dependent (-specific)—and self-regulation, which is comprised

of metacognition and motivation. Metacognition is further composed of self-

monitoring and planning, and motivation is comprised of effort and self-efficacy.

Knowledge maps are reliable and efficient for the measurement of the content

understanding portion of the O’Neil Problem Solving model, and CRESST has

developed a simulated World Wide Web-based knowledge mapping environment to

evaluate problem solving strategies.

Problem solving can place a great amount of cognitive load on working

memory. Instructional strategies have been recommended to help control or reduce

that load. One such strategy is scaffolding. While there are a number of definitions of

scaffolding (e.g., Chalmers, 2003; van Merrionboer et al., 2002; van Merrionboer et

al., 2003), what they all have in common is that scaffolding is an instructional

method that provides support during learning. Clark (2001) described instructional

methods as external representations the internal processes of selecting, organizing,

and integrating. Instructional methods provide learning goals, monitoring, feedback,

selection, hints, prompts, and various advance organizers (Alessi, 2000; Clark, 2001;

100


Jones et al., 1995; Leemkuil et al., 2003). Each of these components either reflects a

form of scaffolding or reflects a need for scaffolding

One form of scaffolding is graphical scaffolding. A number of studies have

reported the benefits of maps, which is a type of graphical scaffolding (Benbasat &

Todd, 1993: Chou & Lin, 1998; Chou et al., 2000; Farrell & Moore, 2000-2001;

Ruddle et al, 1999). While Chou and colleagues (Chou & Lin, 1998; Chou et al.,

(2000) found that certain type of maps (global navigation maps) can benefit search

efficiency, in terms of speed and revisit rates, it is unclear whether map type affects

content understanding. According to Chou and colleagues, map types also do not

appear to affect continuing motivation. Tkacz (1998) found that individual

differences affect map interpretation—particularly one’s orientation and reasoning

ability. Tkacz also found that video game performance is affected by all spatial skills

(particularly orientation and mental rotation). In contrast, Mayer et al. (2002) found

that graphical support aided in content understanding, regardless of spatial ability,

and that use of graphical aids in a graphical task resulted in higher transfer than non-

use of aids or use of other types of aids (e.g., textual or verbal).

While navigation maps can reduce or distribute cognitive load (Cobb, 1997),

they also have the potential to add load, ultimately counteracting their possible

positive effects. The spatial contiguity effect addresses the cognitive load imposed

when multiple sources of information are separated (Mayer & Moreno, 2003) and the

split attention effect, which is related to the contiguity effect, occurs when dealing

with two or more related sources of information (Atkinson et al., 2000). Therefore,

while navigation maps can provide valuable cognitive support for navigating virtual

101


environments, such as computer-based video games, the potential for extra load

caused by split attention must be considered and, where possible, addressed. Mayer

(2001) proposed that the split attention effect can be resolved by placing the

components near each other; for example, placing text labels near their related

imagery in an illustration. In this study, the use of a navigation map separate from

the screen where the game appears is expected to introduce additional cognitive load

—no viable solution was found to resolve this situation in this study.

102


CHAPTER 3

METHODOLOGY

Research Questions and Hypotheses

Research Question 1: Will the problem solving performance of participants

who use a navigation map (the treatment group) in a 3-D, occluded computer-based

video game (i.e., SafeCracker®) be better than the problem solving performance of

those who do not use the map (the control group)?













better performance.

Research Question 2: Will the continuing motivation of participants who

use a navigation map in a 3-D, occluded computer-based video game (i.e.,

103


SafeCracker®) be greater than the continuing motivation of those who do not use the

map (the control group)?




group).

Research Design

This research consisted of two studies: a pilot study and a main study. The

design of the main study was a true experimental posttest only, 2 by 2 repeated

measures design with randomized assignment of participants. It involved two groups

(one treatment group, which used a navigation map, and one control group, which

did not use a navigation map) and occasions (one after the first game and one after

the second game). Each occasion consisted of creation of a knowledge map and

responding to a problem solving strategies questionnaire which elicited both

retention and transfer responses. Participants were randomly assigned to either the

treatment or control group. Group sessions involved only one group type: either all

treatment participants or all control participants. Due to limited availability of

computers, session size was limited to a maximum of three participants. At the end

of the approximately 90 minute session, participants were debriefed and allowed to

continue playing on their own for up to 30 additional minutes (to assess continuing

motivation).

104


Study Sample

University of Southern California (USC) Human Subjects approval was

requested on June 17, 2004. Revisions were requested on July 15 for the recruitment

flyer and the Informed Consent Form. Changes to these two forms were made and

resubmitted on July 22, 2004. On July 26, 2004, the USC Institutional Review Board

(IRB) approved all forms, allowing participants to be contacted and the experimental

sessions to begin.

Pilot study sample. The pilot study sample consisted of two participants and

was conducted September 28 and 29, 2004. The purpose of the pilot study was to

review the procedures and instruments that were to be utilized in the main study.

The sample for the pilot study was a convenience sample and consisted of one

female approximately 49 years 4 months of age and one male approximately 32

years and 8 months of age. Both subjects were graduates of a southwestern

university. Both participants had a reasonable level of computer proficiency,

virtually no video game experience, and no prior experience with the game

SafeCracker©.

Main study sample. Between November 11, 2004 and March 21, 2005,

seventy-one English-speaking adults, ranging in age from 19 years and 4 months to

31 years and 11 months, participated in the main study. The average participant age

was a few days less than 23 years old. All the participants for the main study were

undergraduate students, graduates, or graduate students of a southwestern university.

Solicitation of participants for the main study. Participation was solicited

through several methods. The primary method was a standard paper sized—8 and a

105


half by 11 inch—flyer (Figure 4) posted in various locations within five of the

university’s schools; business, engineering, communication, cinema, and education.

These schools were chosen for a number of reasons, including: how many locations

within their facilities they allowed flyers to be posted at; ease of, or ability to get,

approval to post flyers; a belief by the researcher that their students might be

interested in participating in a video game study.

Figure 4: Participant Solicitation Flyer

Flyers were also sent via email attachment to two of the university’s student

organizations that the researcher believed would include students potentially

interested in this type of study. The organizations were a student video game

development group and a student television and film special effects group. The

researcher was the faculty advisor to the video game group. Flyers were also posted

106


around campus at locations approved for display of announcements. These locations

included student congregation areas, major outdoor pathways, and parking structure

stairwells. The flyer (Figure 4) included a statement that participants would be paid

$15 for approximately 90 minutes of participation and participants must have no

prior experience playing the personal computer- (PC-) based video game

SafeCracker® (Daydream Interactive, 1995/2001). Email contact information was

provided on the flyer.

Randomized assignment for the main study. As email requests for

participation in the study arrived, each participant was randomly assigned to either

the treatment or control group. Participant information was entered, in the order in

which their emails were received, into a Microsoft Excel 2002 spreadsheet for

tracking purposes. The spreadsheet was used for other logistical issues related to the

study. Randomized assignment was accomplished using a random number generator

within a Microsoft Excel 2002 spreadsheet. When the participant’s last name was

entered and either the enter or tab key was pressed on the computer, the random

number generator would display a number between 0.000000000 and 1.000000000,

in increments of .000000001. If the number was from 0.000000000 to 0.500000000,

the participant was assigned to the Control group. If the number was from

0.500000001 and 1.000000000, the participant was assigned to the Treatment group.

Various study times were selected for each group and particpants were sent a list of

those times relevant to their group. Participants responded by listing one or more

times during which they could participate. From the responses, the researcher

scheduled participants to best fill each available time slot.

107


Number of particpants whose data were analyzed. The data from 64

participants were analyzed; 33 from the treatment group (the navigation map) and 31

from the control group (no map). A total of 71 students participated in the study,

with 68 completing the study. Thirty-four of those completing the study received the

treatment, the navigation map, and thirty-four were in the control group and did not

receive the navigation map. The navigation map was a topological (overhead) floor

plan of the game’s playing environment; a mansion. Figure 5 shows the navigation

map used for the first of two games played.

Figure 5: Sample Navigation Map

Those in the treatment group also received instruction on how to read the

map and how to use the map for planning and navigation (see the section entitled

“Introduction to Using the Navigation Map” later in this chapter for information on

the map training). Those in the control group did not receive the navigation map and

were only given brief instruction on how to navigate the mansion without a map (see

108


the section entitled “Script for the Control Group on How to Navigate the Mansion”

later in this chapter for the script administered to the control group).

Of the 71 students that participated in the study, 68 completed the study

(three participants did not complete the study due to computer errors), but the data

from only 64 participants were included for analysis in this study. The main

experiment took place between November 11, 2004 and March 21, 2005. Near the

end of the experimental phase, two of the computers began to exhibit problems. One

computer began to freeze (quit accepting or responding to user input). The other

computer began to intermittently display an error during the second round of game

play during a session. In most cases, turning off the computers before various phases

of the study alleviated or prevented problems. However, near the end of the data

collection phase of the study, the computer that had intermittently been freezing

began to regularly freeze. Three participants who used that computer had to end the

study early and not enough data was collected by either participant to be analyzable

These were the three subjects that had not completed the study (causing the reduction

from 71 participants to 68). From that point onward, that computer was not used in

the study, limiting participation to only two per session. For the other computer that

had been exhibiting problems, restarting the computer at various phases worked well

for all but one participant. This participant had to leave the study early and not

enough data was collected for analysis.

In one session involving two participants, the researcher inadvertently had the

participants overwrite a file with some of their prior data, making the comparison

between the occasion 1 data and the occasion 2 data impossible. Those two

109


participant’s data were not included in the data analysis. This reduced the number

from 68 to 66. The two final participants not included in the data analysis had to

leave very early in the study due to computer problems. In both cases, the

participants had only been shown how to use one software package (the Knowledge

Mapping software). It was determined at the time to be acceptable to have those two

participants return to complete the study at a later date without compromising the

validity of their data. They did return and completed the study. However, after

reconsideration, it was decided that the instruction the two participants had received

the first time they participated made them different than the other participants.

Therefore, their data was not included for analysis. This reduced the number of

participants whose data were analyzed from 66 to 64.

Hardware

The pilot study took place in the home office of the researcher. The computer

utilized for the pilot study was a 450 MHz (megahertz) desktop computer made by

Tiger Direct (http://www.tigerdirect.com) with 64 MB (megabytes) of RAM

(random-access memory), a standard computer keyboard, a 3-button mouse, and a

21” CRT (cathode-ray tube) monitor.

The main study took place in the campus office of the researcher, where three

computers were set up for the study. A table was set up for two of the three

computers to be placed side by side. One of those computers was a Pentium 200,

NeTPower Symmetra computer with 128 MB of RAM that originally ran the

Windows NT® operating system, but was installed with the Windows 98® operating

system for the study. The computer configuration included a standard computer

110


keyboard, a 3-button mouse, and a 17” CRT monitor. The computer placed next to

the NeTPower Symmetra on the table was a Sony PCG-F520, Pentium III laptop

computer with 192MB of RAM, and running the Windows 98 operating system. The

laptop’s keyboard was used, but an external USB (universal serial-bus) 2-button

mouse was added. A 14” CRT monitor was attached to this computer, because the

laptop’s built-in LCD (liquid crystal display) monitor was not very good; the

displayed images were very dark and had low contrast, and the screen exhibited a lot

of reflection and glare, making visibility difficult. The NeTPower Symmetra’s power

case was placed on the desk between the two computers, to reduce the visibility by

participants of each other’s monitor; The researcher was concerned that a participant

might be distracted by the imagery on another participant’s screen.

The third computer was a Dell Latitude D500, Pentium M laptop computer

with 256MB RAM, and running the Windows 98 operating system. As with the other

laptop, this laptop’s keyboard was used, but a serial bus 2-button mouse was added.

This computer was placed on a lateral file cabinet. A monitor was not attached to this

laptop, because its built-in 12” LCD screen produced a satisfactory picture.

The three computers were placed so that participants could not easily see

what other participants were doing and participants had sufficient room to use the

mouse and to write on paper. The primary mode of computer input and interaction

during the study was via the mouse. The only time the keyboard was used was for

entering a file name when saving various types of data. Two files were saved during

one phase (occasion 1) of the study and two files were saved during a second phase

of the study (occasion 2), for a total of four files.

111


Instruments

A number of instruments were included in the study: a demographic, game

play, and game preference questionnaire (see the next section, entitled

“Demographic, Gameplay, and Game Preference Questionnaire), two task

completion forms (see Figures 6 and 7), a self-regulation questionnaire (Appendix

A), the computer-based video game SafeCracker® (see the section entitled

“SafeCracker”), a navigation map of the game’s environment (see Figures 8 and 9),

a problem solving strategy retention and transfer questionnaire (see the section

entitled “Domain-Specific Problem Solving Strategies Measure”; and knowledge

mapping software (see the section entitled “Knowledge Map”).

Demographic, Gameplay, and Game Preference Questionnaire

At the start of the experiment, a questionnaire was administered to elicit

gender, age, amount of weekly video game play, and preferred game types. For

gender, participants marked either the male or female check box. For age,

participants entered both the number of years and the number of months. For amount

of weekly video game play, participants checked one of four boxes: none, 1 to 2

hours, 3 to 6 hours, and greater than 6 hours.

The game types section listed 8 items: Puzzle games, RTS games, FPS

games, Strategy games, Role Playing games, Arcade games, PC games, and Console

games. The first five items in the game types section were game genres; types of

games. The last two items were game platforms; specific combinations of hardware

and software. The sixth item, Arcade games, was both a game genre and a platform;

a type of game or a combination of hardware and software. For each of the game

112


types, participants entered a number from 0 to 5, with 1 indicating low interest and 5

indicating high interest. Participants were prompted to enter a zero if they did not

play that game type or did not know the particular type of game type or what the

initials meant. It was determined by the researcher that those who played RTS or FPS

games, in particular, would know what those terms meant, since the terms were

commonly used by players of those particular game genres; RTS stands for Real-

Time Strategy and FPS stands for First-Person Shooter (also known as first-person

perspective). If a participant asked what a term meant, he or she was prompted to

enter a zero.

The last two game types were gaming platforms. PC games, which stand for

Personal Computer games, refers to games played on personal, or home, computers

(PCs), such as an Apple computer or Windows-based computer. Console games were

those games played on gaming consoles, such as PlayStation®, X-Box®, or

Nintendo® game consoles.

Arcade games referred to both a genre and a platform. As a genre, arcade

games are short, and often rapid reaction, games with short playing durations and

only one or two goals. As a game platform, arcade games historically refers to

games played on large stand-alone gaming consoles like those found in public

arcades. Today, however, arcade style games are also available on home computers

(PCs).

The divisions for amount of weekly game play included in the questionnaire

(none, 1 to 2 hours, 3 to 6 hours, and greater than 6 hours) were based on a study

conducted in 1996 by the Media Analysis Laboratory, Simon Fraser University,

113


Burnaby, British Columbia, Canada. The study surveyed 647 children ranging in age

from 11 to 18, with 80% between the ages of 13 and 15. Six hundred forty six

participants completed the survey (351 male and 295 female). Based on the findings

of this study, which indicated that most children surveyed played video games

between 1 and 6 hours per week, the four divisions used in this study were created.

Information on the British Columbia study can be found at

http://www.mediaawareness.ca/english/resources/research_documents/studies/

video_games/vgc_vg_and_television.cfm

Task completion form

Immediately before the start of each game (the game was played twice during

the study), participants were handed a Task Completion form. Figure 6 shows the

task completion form for the first game of the pilot study. Figure 7 shows the task

completion form for the second game of the pilot study. The task completion forms

served two purposes. First, they provided the researcher with data on which safes

were opened. Second, they provided participants with an advance organizer for tasks

to be completed during each game.

The task completion forms listed the names of the rooms that were involved

in a particular game and the safes that could be found in each room. Players were

told to mark off (check the boxes for) each safe they opened and to be sure to mark

them off as soon as a safe was opened, so as not to forget which safes were opened

during a game. At the end of each game, players were prompted to check the form to

ensure all opened safes were marked off.

114


Figure 6: Task Completion Form 1 for Pilot Study

Figure 7: Task Completion Form 2 for Pilot Study

115


Self-Regulation Questionnaire

A trait self-regulation questionnaire (Appendix A) designed by O’Neil and

Herl (1998) was administered to assess each participant’s degree of self-regulation,

which is one of the three components of problem solving ability as defined by

O’Neil (1999). Reliability of the instrument ranges from .89 to .94, as reported by

O’Neil & Herl (1998). The 32 items on the questionnaire were composed of eight

items each for the four self-regulation factors in the O’Neil (1999) Problem Solving

model (see Figure 1): planning, self-monitoring, self-efficacy, and effort. For

example, item 1 (Appendix A) “I determine how to solve a task before I begin.” is

designed to assess a participant’s planning ability; and item 2 “I check how well I am

doing when I solve a task” was to evaluate a participant’s self-monitoring. The

response format for each item was a Likert-type scale with four possible responses;

almost never, sometimes, often, and almost always. The self-regulation form was

administered in printed format, with participants using either a pen or pencil to enter

responses (a number from 1 to 4) for each question. Responses were later entered

into a Microsoft Excel 2002 spreadsheet and totals for each for the four self-

regulation factors were generated using Excel’s SUM function.

SafeCracker

The non-violent, PC-based video game SafeCracker® (Daydream Interactive,

1995/2001) was selected for this study, as a result of a feasibility study by Wainess

and O’Neil (2003). The purpose of the feasibility study was to recommend a video

116


game for use as a platform for research on cognitive and affective components of

problem solving, based on the O’Neil (1999) Problem Solving Model (see Figure 1).

According to Wainess and O’Neil (2003), a primary factor for selecting

SafeCracker® was time constraints. During the feasibility study, it had been decided

that participants should not be required to spend more than one and a half hours in

any study for which the game would be used. In addition, it was desirable to include

multiple iterations of gameplay within that time period. With SafeCracker, players

would be able to learn the controls and interface and enter the main game

environment (a mansion) in approximately 15 minutes. Using only two or three of

the mansion’s approximately 50 rooms could provide a large enough set of tasks, in

the form of clues and objects to find and safes to open, to examine complex problem

solving in 10 to 20 minutes, allowing for multiple problem solving tasks using

different room combinations. Table 4 replicates the same list of game characteristics

as found in Table 1 but adds a final column indicating the characteristics of

SafeCracker. From Table 4, it can bee seen that SafeCracker met the characteristics

of a simulation-game: It met most of the characteristics of a game; however, it

included elements of a simulation. It contained the simulation element of cause-

effect relationships through its puzzle designs. It contained a goal structure that could

be considered primarily non-linear, as with simulations. And it did not contain the

game element of constraints, privileges, and penalties. Therefore, while meetings

most of the characteristics of a game, it included characteristics that were decidedly

those of a simulation, making SafeCracker a simulation-game.

117


Table 4Characteristics of Games, Simulations, and SafeCracker

Characteristic Game Simulation SafeCrackerCombination of ones

actions plus at least one other’s actions

Yes (via human or computer)

Yes Yes (via computer puzzles)

Rules Defined by game designer/developer

Defined by system being replicated

Defined by game designer/developer

Goals To win To discover cause-effect relationships

To win

Requires strategies to achieve goals

Yes Yes Yes

Includes competition Against computer or other players

No Against computer

Includes chance Yes Yes YesHas consequences Yes (e.g., win/lose) Yes YesSystem size Whole Whole or Part WholeReality or Fantasy Both Both RealitySituation Specific Yes Yes YesRepresents a

prohibitive environment (due to cost, danger, or logistics)

Yes Yes Yes

Represents authentic cause-effect relationships

No Yes Yes

Requires user to reach own conclusion

Yes Yes Yes

May not have definite end point

No Yes No

Contains constraints, privileges, and penalties (e.g. earn extra moves, lose turn)

Yes No No

Linear goal structure Yes No NoLinear intervention No No NoIs intended to be

playfulYes No Yes

118


The plot of SafeCracker is that the player is a highly trained security

specialist applying for a position as head of security at Crabb and Sons, a prestigious

security company. The company’s primary business is to manufacture custom made

safes ranging from fairly standard box safes to deceptive and complex hidden safes.

As part of the job application, the player must break into the premises of Crabb and

Sons, a mansion, during the night and navigate the building to find and open 34

safes. And the player has only 12 hours to do it.

To open some of the safes, the player must collect clues such as wiring

diagrams, tools such as keys, and other objects such as a cassette tape. Some safes

are easily solved through trial and error. Other safes require prior knowledge, such as

knowing who Lafayette was. Several of the safes chosen for this study could be

opened via trial and error, some needed clues, keys, and/or other objects, one

required prior knowledge (needing the word Lafayette to be entered), and one

required an understanding of math sequences or prior knowledge of Pascal’s

Triangle (see http://mathforum.org/workshops/usi/pascal).

Navigation map

Gameplay in SafeCracker takes place in a two story mansion. For the

purposes of this study, three rooms on the first floor were utilized for each of two

games involved in the study. The two games had one room in common, for a total of

five different rooms. A navigation map, in the form of a topological floor plan of the

first floor of the mansion, was downloaded from http://www.gameboomers.com/

wtcheats/pcSs/Safecracker.htm. The navigation map was subsequently modified

using Adobe® Photoshop® 6.5, to alter the view of the navigation map from one-

point perspective to a flat 2-D image, to remove unnecessary artifacts, to remove

room numbers for each room, to add the appropriate name to each room in

119


accordance with names displayed on the game’s interface, and to add a compass

symbol to the top right side of the map. Two navigation maps were created. Figure 8

shows the final, modified version of the navigation map used for game 1. Figure 9

shows the final, modified version of the navigation map used for game 2. For each

map, three rooms were also darkened using Adobe® 6.5, to represent the three rooms

containing the safes needing to be opened in those games.

Figure 8: Navigation Map for Game 1

Figure 9: Navigation Map for Game 2

120


Based on the work of Chou and Lin (1998), these maps (Figures 8 and 9)

would be considered a global navigation map. There maps are considered global

because they provides information on the whole of the environment; all rooms on the

floor. A local map would have shown the details of a particular room, such as the

locations of furniture and safes. A local map might also have shown the locations of

objects and clues within the room necessary for opening the safes.

The three darkened rooms in each navigation map represented the three

rooms involved in each game. The three darkened rooms were included on the maps

handed to the participants and participants were informed that, as explained during

the navigation map training session (discussed later in this document), those rooms

represented the three rooms that contained the safes that were to be opened and all

the clues and items needed for opening the safes. Also notice that one room, the

Technical Design Room, was included in both games. The various rooms of the first

floor were examined by two researchers, to determine the best set of rooms to use for

this and two other studies (see Chen, 2005; Shen, in preparation). The considerations

were (A) to choose three rooms for each trial, (B) to have the safes require the least

amount of domain-specific prior knowledge beyond knowledge that every university

student should know, (C) and to ensure that all clues and other objects needed for

opening the required safes were contained within those rooms.

For the first game, the three rooms selected were the Reception Room, which

was the room used during the training session. One safe in that room was opened by

the participants during training. The other safe in the Reception Room had a

programming flaw that rendered the safe unable to be opened at times. Therefore, for

121


the first game, participants began by opening a game already in progress. In that

game, the two safes from the Reception Room were already opened, the contents

appeared in the game interface’s inventory section, ready to be used when necessary,

and the participants were located in the Reception Room facing north. The direction

was an arbitrary decision by the researcher. Because the game’s interface contained a

compass that participants could use to help with navigation, the direction was told to

the participants to prime them to use the compass.

Participants were told that the safes from the Reception Room were opened

and they were directed to examine the items in their inventory. Participants were not

told why the safes were already opened; they were not told about the flaw in the

game’s programming for one of the Reception Room’s safes. Before beginning to

play the game, participants were reminded to search for clues and to write down any

information they deemed important.

For the second game, it was determined by the researchers that the three best

rooms, taking in consideration the rooms already visited for the first game, would

include the Technical Design Room, which had been used in the first game. For the

second game, participants also began by opening a game already in progress. In this

game, the safes from the other two rooms from the first game, the Reception Room

and the Small Showroom, were already opened and their contents placed in the

participant’s inventory, ready for use. The participants began this game in the

Technical Design Room; they were facing north; once again, this direction was an

arbitrary decision by the researcher and participants we told the direction to prime

them to use the interface’s compass.

122


Participants were told that the safes from the other two rooms were opened.

They were also directed to their inventories and told that all the contents from the

safes from those two rooms were in their inventory. Participants were also told that,

even if they had already opened the safes in the Technical Design room during the

first game, they would need to open those safes again. Each of the safes in the

Technical Design room had been opened by 20 participants in the first game. Twelve

participants had opened both safes in the first game. Before beginning to play the

game, participants were reminded to search for clues and to write down any

information they deemed important. It was also suggested that they might want

revisit the Reception Room and the Small Showroom, if they had not looked at all

the clues in those rooms during the prior game.

Knowledge Map

In this study, participants were instructed to create a knowledge map using a

computer-based software program, to evaluate their problem solving content

understanding after playing SafeCracker. According to Plotnick (1997), a knowledge

map, referred to as a concept map by Plotnick, is a “graphical representation where

nodes (points or vertices) represent concepts, and links (arcs or lines) represent the

relationships between concepts” (p. 81). He also commented that the concepts and

links are labeled on the map and the links could be unidirectional, bi-directional, or

non-directional.

During the study, participants played SafeCracker twice and completed a

knowledge map after each game session. The computerized knowledge map used in

this study had been successfully applied to other studies (e.g., Chuang, 2003; Chung

123


et al., 1999; Hsieh, 2001; Schacter et al., 1999). Appendix B lists the knowledge map

specification used in this study (adapted from Chen, 2005). The Knowledge Map

used in this study only offered unidirectional links, but links could be added in each

direction to create bi-directional relationships between concepts.

Content understanding measure. Content understanding measures were

computed by comparing the semantic content score of a participant’s knowledge map

to the average semantic content score of three subject matter experts. According to

Mayer (2003), semantic knowledge refers to a person’s “factual knowledge about the

world” (p. 15). Therefore, the semantic content score derived from the knowledge

map represented a participant’s factual understanding of the concepts and

propositions involved in the game SafeCracker. The experts for this study were

researchers from a prior study involving knowledge map creation when playing the

game SafeCracker (Chen, 2005). Three expert knowledge maps were created for that

study and were used in this study. Figures 10, 11, and 12 show the three expert

SafeCracker knowledge maps.

124


Figure 10: Expert SafeCracker Knowledge Map 1

125



126



The three expert maps (Figures 10, 11, and 12) are based on the general

concepts and propositions relevant to problem solving in the game SafeCracker as a

whole, not the concepts and propositions specific to a room or a safe. For a

description of the process involved in determining and creating the three expert

knowledge maps, see Chen (2005).

Figure 13 shows a sample of a portion of a knowledge map that might have

been created by a participant during this study. It contains four concepts (key, safe,

catalog, and clue) and unidirectional links from key to safe, safe to key, safe to clue,

and catalog to clue. The specific nature of each link is displayed along the link’s

path. For example the link from key to safe included the phrase used for, indicating

127


the proposition of “key used for safe.” The concepts are read in the direction of the

arrow and the text of the link is placed between the text of the two concepts.

Figure 13: Sample Participant Knowledge Map for the Game SafeCracker®

Scoring the knowledge map. The following describes how the participant’s

knowledge maps were scored. A semantic score was calculated based on the

semantic propositions—two concepts connected by one link in the experts’

knowledge maps. Every proposition in a participant’s knowledge map was compared

against each proposition in the three experts’ maps. A match was scored as one

point. The average score of the three expert comparisons would be the semantic

proposition score for the student map.

An example of how to score a knowledge map of SafeCracker is shown in

Table 5. Table 5 contains the scoring data extracted from the knowledge map

disiplayed in figure 13 above.

used forkey safe

catalog

clue

contains

requires

contains

128


Table 5An Example of Participant Knowledge Map Scoring Concept 1 Links Concept 2 Expert 1 Expert 2 Expert3

Key Used for Safe 1 1 1Safe Requires Key 1 1 0Catalog Contains Clue 1 0 1Safe Contains Clue 0 0 1

Final score = total points ÷ number of experts = 8 ÷ 3 = 2.67

The following describes how the data in Table 5 were scored. First, the scores

for the semantic propositions (two concepts plus their link, such as ‘key used for

safe’) were calculated based on whether the same semantic propositions, appeared in

each of the three expert maps. Each time there was a match, the participant’s

semantic proposition was scored with one point. If there wasn’t a matching semantic

proposition in an expert’s map, the participant’s semantic proposition received a

score of zero. Therefore, the score a participant could receive for a semantic

proposition ranged from zero (finding no matching proposition in any expert map) to

three (finding a match in all three expert maps).

The average score of the results of comparison to all three expert maps

became the participant’s semantic proposition score across all three expert maps

would be the semantic score of the participant’s map, that is, the total score from all

three expert comparisons was divided by three. If for example, all three experts were

matched, for a total of three points, the semantic proposition score the participant

received would be one; 3/3 = 1. If the participant had matched only two experts, the

participant would have received .66; 2/3 = .66. If the participant had matched only

one expert, the participant would have received .33; 1/3 = .33. And if the participant

had not matched any expert, the participant would have received zero; 0/3 = 0. The

129


final score for a knowledge map would be determined by adding all the matches

received and dividing the total matches by three. Table 5 above shows that the four

participant semantic propositions matched expert maps eight times. The total of eight

was then divided by three giving the participant a knowledge map score of 2.67.

Domain-Specific Problem Solving Strategies Measure

In this study, a problem solving instrument successfully employed by Richard

Mayer and Roxanna Moreno (see for example, Mayer, 2001; Mayer & Moreno,

1998; Mayer et al., 2003; Moreno & Mayer, 2000, 2004) was modified to measure

the domain specific problem solving strategies of the game SafeCracker. In one

study, Mayer and Moreno (2003) measured retention by having participants respond

to an opened-ended domain-specific statement, “Please write down an explanation of

how lightening works.” Acceptable answers, referred to as idea units, were defined

by the researchers.

An idea unit is a proposition. Brunning et al. (1999) defined a proposition as

“the smallest unit of meaning that can stand as a separate assertion” (p. 54).

Brunning et al. further asserted that propositions are more complex than the concepts

they include. According to Brunning and colleagues, while concepts are relatively

elemental categories, “propositions can be thought of as the mental equivalent of

statements or assertions about observed experience and about the relationships

among concepts. Propositions can be judged to be true or false” (p. 54). These

descriptions of a ‘propositon’ support the earlier definition of a ‘semantic

proposition’ as two concepts plus their link, such as ‘key used for safe.’ In

accordance with Brunning et al.’s (1998) definition and descriptions, this statement

130


is more complex than the concepts it includes, it’s a mental equivalent of an

assertion, and it can be judged as true or false.

Participants’ responses (i.e., idea units) in the Mayer and Moreno (2003)

study were then compared to the experts’ idea units and considered acceptable if the

content matched, regardless of exact wording. Idea units defined by the researchers

included “air rises,” “water condenses,” “water and crystals fall,” and “wind is

dragged downward.” Each response a participant wrote that matched an idea unit

received one point. Retention scores were determined by totaling the number of

matches, with a higher score indicating higher retention.

In the same study by Mayer and Moreno (2003), participants were given the

transfer question, “Suppose you switch on an electric motor, but nothing happens.

What could have gone wrong?” For this question, the researchers generated a list of

acceptable idea units (answers) such as “the wire loop is stuck,” or “the wire is

severed or disconnected from the battery” and participants’ responses were

compared to these idea units. As with the retention responses, one point was given to

each response that matched one of the researchers’ idea units, regardless of wording.

A participant’s transfer score was the sum of the matches, with higher scores

indicating greater transfer.

The problem solving strategy questions designed for this dissertation research

were one retention question and one transfer question relevant to the problem solving

tasks in SafeCracker of finding rooms and opening safes. Table 6 lists the problem

solving strategy transfer retention and transfer questions.

131


Table 6Problem Solving Strategy Retention and Transfer QuestionsQuestion Type Question

Retention List the ways you found rooms and opened safes.Transfer List some ways to improve the design of the game play for

opening safes.

Participants were given four sheets of paper clipped together. At the top of

page one was the retention question “List the ways you found rooms and opened

safes.” Following the retention question were forty-two double-spaced and number

lines spanning to the end of page two. At the top of page three was the transfer

question “List some ways to improve the design of the game play for opening safes.”

Following the transfer question were forty-two double-spaced and numbered lines

spanning to the end of page four.

Using the logic for a related type of transfer test administered by Mayer,

Sobko, and Mautone (2003), the transfer question constitutes a transfer test because

the participants must select and adapt what was learned in the game to fit the

requirement of the question. Instead of simply being cued to recall what had

occurred, which is the function of the retention question, participants had to judge

which aspects of the game and game play were relevant to the question and had to

determine how to link that information to their responses to the transfer question. In

short, the transfer question required the participants to go beyond simply recalling

the game and game play experience, although recalling relevant portions of the game

and game play were certainly a component of the solution.

To answer the transfer question, participants had to recall what they had done

in the game. Next they had to consider what events could have been improved by

132


modifying the game in some way. Then they needed to consider exactly how the

game functioned and determine what was present that shouldn’t have been present or

what wasn’t present that should have been. For example, one participant’s response

to the transfer question was, “make dials bi-directional.” Another participants’

response was, “… a little indicator of a person moving along with some kind of a

map should be given.” A third participant responded, “If clue was already used,

remove from list.” All three responses represent functionality or features that did not

exist in the game. Participants had to infer the benefit of their addition by what was

or wasn’t already in the game and by their gaming experience. Therefore, their

response represented a form of transfer.

As with the Mayer and Moreno (2003) study, each participant response in

this study was extracted into an idea unit and scored against expert idea units. Two

sets of expert idea units were used in this study; one set for the problem solving

strategy retention question and one set for the problem solving strategy transfer

question. The expert idea units for the problem solving strategy retention question

were developed by three researchers, through a three step process. First, each

researcher reviewed a set of problem solving strategy retention idea units created by

Chen (2005) for a study that also used the game SafeCracker. The problem solving

strategy retention question for that study differed from the retention question for this

study. The retention question for the Chen study was, “Write an explanation of how

you solve the puzzles in the rooms” while the retention question for this study was

“List the ways you found rooms and opened safes.”

133


The Chen retention question centered around opening (solving) safes (the

puzzles), while the retention question for this study centered around finding rooms in

addition to opening safes. After reviewing the list from the Chen study, each expert

independently generated a list of idea units applicable to the problem solving

strategy retention question for this study. Then, through discussion among the three

researchers of the various idea units that had been generated, a single list was created

representing the agreed upon 28 idea units for this study. Table 7 lists the problem

solving strategy retention idea units for this study.

Table 7Idea Units of the Problem Solving Strategy Retention Question

1 Scan, observe, analyze, recognize, and/or compare rooms and/or room features.

2 Scan, observe, analyze, recognize, and/or compare safes and/or safe features.

3 Walking and/or turning.4 Search for rooms and/or safes.5 Search for clues, hints, keys, tools, or other objects.6 Recognize or examine clues, hints, keys, tools, or other objects.7 Find or pick up clues, hints keys, tools, or other objects.8 Use clues, hints, keys, tools, or other objects.9 Attempt to open safes through trial and error.

10 Attempt to open safes through organized/methodical method.11 Use interface’s room indicator.12 Use interface’s compass.13 Use map/floor plan.14 Remember items, clues, and/or hints.15 Remember diagrams/images, such as safe solutions or map.16 Draw images/diagrams and/or jot down notes.17 Recognize and/or interpret feedback18 Trial and error/Guessing.19 Apply elimination or methodical method.20 Figure out the direction to a room or safe.21 Plan before doing.22 Determine what the problem and/or difficulty is.23 Determine safe’s procedure, pattern, system, or sequence.

134


Table 7 (continued)Idea Units of the Problem Solving Strategy Retention Question

24 Evaluate effectiveness and/or result of current/previous strategy, and/or change strategy.

25 Make connection between clues and/or hints.26 Apply subject knowledge, such as math or science.27 Use real-life and/or game-related common sense/knowledge.28 Use logic.

The expert idea units for the problem solving transfer question for this study

were developed by three researchers, through a three step process. First, each

researcher reviewed a set of problem solving strategy transfer idea units created by

Chen (2005) for a study that also used the game SafeCracker. The problem solving

strategy transfer question for the Chen study differed from the transfer question for

this study. The transfer question for the Chen study was, “List some ways to improve

the fun or challenge of the game” while the transfer question for this study was “List

some ways to improve the design of the game play for opening safes.”

The Chen (2005) transfer question centered around the “game” as a whole,

while the transfer question for this study centered around the “opening safes” portion

of the game. However, the process of opening safes did include the need to find the

safes as well as the need to find and collect relevant clues and tools. Also, the Chen

transfer question involved improving fun or challenge. The transfer question for this

study involved improving game play, which is less specific than “fun” and

“challenge,” but could include both fun and challenge.

After reviewing the list of expert transfer idea units from the Chen study,

three experts independently generated a list of idea units applicable to the problem

solving strategy transfer question for this study. Then, through discussion among the

135


three researchers of the various idea units that had been generated, a single list was

created representing the agreed upon 21 idea units for this study. Table 8 lists the

problem solving strategy retention idea units for this study.

Table 8Idea Units of the Problem Solving Strategy Transfer Question

1 Add new rules to the game.2 Modify the amount or kind of in-game help, advise, instruction, and/or

demonstration.3 Create new ways of giving in-game help to the players.4 Modify the amount or type of pre-game help, instruction, and/or

demonstration.5 Create new ways of giving pre-game help, instruction, and/or

demonstration.6 Modify the complexity, patterns, or procedures for opening safes.7 Create new safe types, safe features, or methods for opening safes.8 Modify the complexity or procedures for finding rooms or safes.9 Create new room features.

10 Modify the complexity or procedures for finding clues, tools, objects, and/or hints.

11 Create new clue, tool, objects, or hint features.12 Modify existing functionality/features in the user interface (helpers,

tools, interface elements, controls, etc.). 13 Create new interface features (helpers, tools, interface elements,

controls, etc.).14 Increase the connection between rooms, safes, clues, and/or objects.15 Increase the amount, type, and/or function of audio used in the game.16 Create more opportunities for interaction with the game. 17 Modify the background story elements of the game to be more

meaningful and/or interesting.18 Modify the time allotted for the game or game elements.19 Modify existing elements in the game to alter the complexity of the

problem solving experience.20 Create new elements to the game to alter the complexity of the problem

solving experience.21 Add other players in the game to compete or cooperate.

Scoring of the Problem Solving Strategies Retention and Transfer Reponses.

Scoring of the problem solving strategy retention and transfer responses was

a three step process. In step one, two researchers independently reviewed each

136


participant response to determine if a single response represented more than one idea

unit. If it did, the response was divided into multiple responses, each containing a

single idea unit. Then the two lists of responses were compared. With an original

total of 1448 participant responses comprised of both the problem solving strategy

retention and problem solving strategy transfer responses, by breaking some

responses into multiple idea units, the new list contained 1513 responses; an addition

of 65 responses. The two researchers had agreed on all but 38 of those additions,

therefore, agreeing on 1475 out of 1513 idea units, which represented 97.5%

agreement. Next, the two researchers examined each of the 38 discrepancies and

reached agreement on whether each was or wasn’t a separate idea unit. This process

resolved all discrepancies and resulted in the addition of seven more idea units, for a

total of 1520 idea units.

Next, independently, the two researchers assigned an expert idea unit to each

of the 1520 participant responses, assigning expert retention idea units to the

participants’ retention responses and expert transfer idea units to the participant’s

transfer responses. Then, the two lists were compared. The two researchers had

agreed on 1178 of the 1520 idea units, which was 77.5% agreement. As with the

prior process, the two researchers reviewed and resolved each of the 342

disagreements. Since the two lists of participant responses and related idea units now

matched, one list was removed leaving just one list of problem solving strategy

retention and transfer responses with their related idea units.

137


Procedure for the Pilot Study

There were two pilot study participants. By flip of a coin, one participant

was randomly assigned to the treatment group (the navigation map) and the other

participant was assigned to the control group (no map). The pilot study was

conducted one participant at a time. Each of the two studies (one treatment and one

control) took approximately 91 minutes to administer and began with introducing the

participant to the objective of the experiment, describing the experiment as an

examination of methods that might help student performance when using a video

game for learning, but not discussing the issue of navigation maps. The introduction

took approximately three minutes. Next participant and the researcher signed a

consent form and the participant was assigned a three-digit number that had been

randomly generated prior to the study. The three-digit number was used for

confidentiality purposes; by assigning a number to each participant, all that

participant’s data would be associated with a number, not a name.

Administration of Demographic and Self-Regulation Questionnaires.

Following the brief three minute introduction, participants were asked to fill

out the demographic and self-regulation questionnaires. See the earlier sections

“Demographic, Game Play, and Game Preference Questionnaire” and “Self-

Regulation Questionnaire” for complete descriptions of the items contained in the

two questionnaires. Participants were told they would have eight minutes to fill out

the questionnaires.

138


Introduction to Using the Knowledge Mapping Software.

Following administration of the demographic and self-regulation

questionnaires, participants were introduced to the knowledge mapping software.

Ten minutes of the study were allocated to this process. Participants were asked to

start the knowledge mapping software. Once started, knowledge mapping was

explained. Participants were told that knowledge mapping involved concepts and

links. It was explained that a concept was an idea or word and could represent

something concrete like house or something abstract like love. It was then explained

that two concepts could be linked based on some sort of relationship. Relationships

included causal relationships, temporal or chronological relationships, or simple

relationships; indicating ways in which the two concepts were connected.

Examples of all three relational types were given, by selecting examples

from several random domains, such as the causal relationship of ‘learning leads to

knowledge.’ The components of each relationship were explained. In the example of

‘learning leads to knowledge,’ it was explained that learning and knowledge were the

concepts and the phrase ‘leads to’ was a causal connection between the two

concepts; In other words, learning causes knowledge. Next it was explained that

research had found that a person’s ability to create a knowledge map of a domain

was directly related to that person’s understanding of the domain; the more accurate

and complete the knowledge map, the greater that person understood the domain.

Then the Knowledge Mapping interface was explained by describing the

function of the ‘Add Concepts’ menu item and the three on-screen buttons (see

Figure 2). Participants were asked to click on a concept to add it to the screen.

139


Participants were then prompted to move the concept around. Next participants were

asked to add a few more concepts.

Participants were then asked to click the Link button and told that would

switch them to a mode that would allow links to be created between concepts.

Participants were told to click and drag from one concept to another concept. That

caused a dialog box to open. Participants were prompted to click on the dialog box’s

pull-down menu, to see a list of available links. Participants were then prompted to

click on a link, which caused the dialog box to close and an arrow to be drawn from

one concept to the other, with the link text they had selected appearing along the link

arrow’s path.

Participants were asked to create several more links, after which they were

told to click the third mode button, the Erase button. Next, participants were asked to

click on the words of a link and saw the link disappear. Next they were prompted to

click on a concept that had at least one link connected to it and watched as both the

concept and its links disappeared. Participants were reminded that once they entered

a mode (Move, Link, Erase), all they could do was that mode. And they were told

there was no undo button on the software. So if they accidentally deleted a concept

and, therefore, all links going to or from that concept, they would need to recreate

the concept and all its links. It was suggested they change to link or move mode, as

soon as they were done erasing items, to prevent any unwanted erasures. Participants

were asked if they understood how to use the software. Upon receiving a positive

response, they were shown how to exit the software.

140


Introduction of the Game SafeCracker

Participants were told they would next learn the game, Safecracker, and were

prompted to open SafeCracker by clicking on an icon on the desktop. Over the next

15 minutes, participants were guided through entering the game, finding the mansion

(the main game play area of SafeCracker), entering the mansion, searching the first

room, and opening one safe. During this 15 minute period, participants using the

navigation map (the treatment group) were also taught to read the navigation map

and to plan and find paths. The navigation map group was also given some strategies

for playing the game (see the next section on “Introduction to Using the Navigation

Map”). For navigation and strategy instructions given to the control group, see the

section “Script for the Control Group on How to Navigate the Mansion.”

To ensure equivalent training on using SafeCracker, all participants received

the same instruction, by use of a script. The next paragraph begins the script that was

used for the pilot study. As will be discussed later in the “Adjustments to the

SafeCracker instructions” subsection under “Results of the Pilot Study,” a number of

changes were made to the SafeCracker training script, as a result of observations,

discussions, and participant comments that occurred during and after the pilot study.

See the script under the main study section of this dissertation, to see the final

version of the script after modifications were made based on feedback from the pilot

study. Note: In the script, the term beat, which appears in the script, is a common

term in script writing and refers to a “momentary pause in dialog or action” (Armer,

1988, p. 260). The term long pause does not have a history in script writing and is

used here to indicate a pause of at least two seconds. Most text in parentheses,

141


including ‘beat’ and ‘long pause’ are notes to the researcher as reminders or cues

during delivery of the script. Text in all uppercase letters were cues to deliver that

text with greater emphasis than other text. As an exception, text in all uppercase

letters, but in parentheses, were reminders to the researcher.

SafeCracker training script. Thank you for participating. In this study, you

will be asked to accomplish a series of tasks. The tasks will be to locate and open

various safes in various rooms. In order to open some of these safes, you will need to

find certain items. You will be told which rooms to visit. Those rooms contain all the

items needed and all the safes you will need to open. You do not need to spend time

in any other rooms. Even though the mansion has two floors, all the rooms you will

visit are on the first floor. Do not go to the second floor.

Your goal is to open all the safes in the rooms you are given. For each room,

you will be told the room’s name (e.g., the Small Showroom). Together, we will

walk through the steps needed to find and enter the mansion. Then, we will walk

through searching the first room and opening one safe. After that, you will given the

number and name of several rooms and will be required to find the rooms and open

the safes. Let’s work our way into the mansion.

GETTING INTO THE MANSION: You see the game’s start menu with four

main buttons. Don’t do anything until I tell you to. Once I tell you to click the

“new” button to begin a new game, you’ll see the game’s main interface screen and a

phone. The phone will be ringing. As move your cursor to the top part of the phone’s

hand piece, you’ll notice that cursor symbol is a double circle. When you’re over the

part of the phone piece you can click, the cursor will turn into a double circle with a

142


hand. This hand symbol indicates you’re over something you can grab. You will then

click on the phone with the left mouse button. Before you click the hand piece, be

sure you’re prepared to listen carefully to the message. It only plays once. Also,

you’re going to need to write down a four-digit number. Have paper and pencil

ready. Once the message is complete, you will click on the phone piece hook to hang

up the phone. Once you’ve hung up the phone, music will begin playing, to turn it

off, click the off button on the right side of the screen. Go ahead, now, and click the

“new” button.

(Once everyone’s done listening to the message) Click in the large center

screen and, while keeping the left mouse button depressed, move the mouse left and

right. The scene will pan left and right. If you stop moving but continue to hold the

mouse button down, the scene continues to pan. The wider you moved the mouse,

the faster the scene will pan. You can also use the left and right cursor arrows on

your keyboard to pan left and right; try that. In addition, you can move the mouse up

and down or use the up and down arrow keys to tilt your view upward or downward.

Now let’s exit the phone booth. Rotate until you see the phone booth door,

then click to open the door. Once the door is open, you can click to move outside the

phone booth. The cursor symbol that indicates you can move forward is a double-

circle with an upward facing arrow. Once outside the phone booth, rotate until you

see the lit two story mansion across the street. Now listen to my next series of

instructions before doing anything. You need to move down the street to the

crosswalk just before the mansion. Then cross the street. Next, you’ll click a couple

143


times to move along the sidewalk until you’re in front of the mansion’s gate. Go

ahead now and move to the mansion’s front gate.

Move your cursor until you get the circle and hand symbol, when you point

to the small lock on the center of the gate. Then click. This locks you into a close up

view of the three tumblers on the lock. Each contains a symbol. Go ahead a take a

moment to try opening the lock. You can rotate the tumblers by clicking on them.

(Wait one minute). If you haven’t opened the lock yet, set the three tumblers to

music symbols and the lock will open. Then move to the front door. You’ll need to

navigate around the fountain to get to the front door. Once you’re at the front door,

click on the keypad box to the left side of the door. Then click on the appropriate

buttons, to enter the four digit code you wrote down at the phone booth. Once the

code is accepted, you can click on the door to open it and then click the inner door to

open it as well. Then click to move into the mansion. Go ahead and take a few

seconds to look around and move around the room you’re in. Do not leave the room.

(Wait 15 seconds) Now let’s collect some objects. Navigate around the desk

until you’re facing the computer on the right. It’s extremely important that you do

not click on anything unless I tell you to. (Wait for everyone to get to the correct

position). Click on the blue coffee mug. The item shows up in the small left viewer,

where you can rotate it. Next, click on the piece of paper to the left of the blue cup. It

contains some diagrams. If you move your cursor toward the bottom of the paper, a

down arrow symbol appears. Click it to see more of the bottom portion of the paper.

You can move the cursor to the top and click to return to the top portion of the paper.

144


Click the back button to exit viewing the paper. Find the two other pieces of paper on

the desk. Only one can be clicked. Now, let’s open a safe.

On either side of the front wall of the room are safes. The left side has a

brown and gold safe, the right side a blue safe. Move to the blue safe. Click on the

safe to lock yourself onto the safe. To exit the safe, click the BACK button on the

right side of the screen. Go ahead and try that, then click to lock yourself back onto

the safe. To open the safe, you need to set the three dials to the correct numbers. Set

the three dials, then click the safe handle. The three lights will either flash green or

be a steady green. The lights from top to bottom represent the three dials from left to

right. If you select the correct number, the light will be a steady green. Once all three

lights are a steady green, the safe will open. Go ahead and open the safe. Before

leaving the safe, be sure to click on each of the objects in the safe, to add them to

your inventory. Once you leave the safe, you cannot reopen it.

NOW, SHOW THE IMPORTANT INTERFACE COMPONENTS (E.G.,

THE ROOM NAME INDICATOR).

FOR MAP USERS, READ THE “SCRIPT FOR INTRODUCING MAP TO

PARTICPANTS”

FOR NON-MAP USERS, READ THE “SCRIPT FOR THE CONTROL

GROUP ON HOW TO NAVIGATE THE MANSION”

THEN, ANNOUNCE THE FIRST TASK AND THE ROOMS INVOLVED.

FOR THE MAP USERS, HAND OUT THE APPROPRIATE MAP.

145


Introduction to Using the Navigation Map.

Those in the navigation map group (the treatment group) were next

introduced to reading the navigation map and were given instruction on path finding

and path planning. They also received instructions on strategies for playing the

game. To ensure equivalent learning by all those in the treatment group, a script was

utilized (see below). Those in the control group were given simple guidelines for

navigation. See the next section entitled “Script for the Control Group on How to

Navigate the Mansion” for information on the script given to the control group.

To support the navigation map training script, a special version of the

navigation map, a training map, was created (Figure 14), displaying a portion of the

floor plan, along with shaded portions, labels, and arrows, as aids to the script.

The training map was handed to each participant on a standard sheet of white

paper and contained the title “How to read the map.” As participants viewed the

training map, the script was read. In addition to training on map reading, path

planning, and path finding, the script included some strategies for playing the game.

The script also contains two words or phrases in parentheses: beat and long pause.

The term beat is a common term in script writing and refers to a “momentary pause

in dialog or action” (Armer, 1988, p. 260). The term long pause does not have a

history in script writing and is used here to indicate a pause of at least two seconds.

The following script was read to the navigation map group.

146


Figure 14: Training Map

Training map script. This is a map of the first floor of the mansion. You will

use this map to help navigate to the various rooms. Currently, you’re in, the

reception room, the large room in the middle of the bottom portion of the map. The

map shows all the rooms on the first floor with their related names. These will match

the names of the rooms that contain the safes you will be asked to open and the items

that will help you to open the safes. The names also match the names that will appear

in the name indicator on your interface, which you have already been shown. You

will not need to visit any other rooms, unless they are along a path you take in order

to get to a required room. In addition to the room names, the map also shows the

locations of the doors in each room. If you need to, you are allowed to write on this

map.

Let’s take a moment to learn how to read the map and use the map. This map

shows a portion of the bottom floor and includes text labels describing of the most

important map features. On the left side are four labels. The top label on the left

147


(point to the label) contains the words “room name” and points to the name of the

room entitled “Big Showroom.” Take a look and you’ll see that every room has a

label. Those areas that do not have names are either closets or bathrooms. For each

of your two tasks, you will be told the names of the rooms you must visit. As shown

to you earlier, there is a room name indicator on the interface. You will use this

indicator to determine which room you are in.

The middle label on the left contains the word “stairs” and points to a block

of black and gray stripes. That pattern indicates stairs. Notice that there’s another set

of stairs just to the right and a small set of stairs connecting the two (point to these

features).

On the left side, near the bottom is a label with the word “door.” Gaps or

open spaces between rooms indicate doors. Every room has at least one door and

most have several doors. (Point to several door openings.)

On the left side, at the bottom is a label with the words “Main Entrance” and

an arrow pointing to the door you came through to enter the mansion. This is the

only door on the map that is not indicated using an opening or gap. (Point to the

door.)

In the middle of the map is a label with the word “toilet” (point to the label).

The arrow points to a small circle, which is the symbol for a toilet (point to the

circle). There are other bathrooms in the mansion that have toilets, but for some

reason, the people who created this map chose to only show this toilet.

On the right side of the map are three labels. The one on the far right side of

the map and containing the words “points north” points to a symbol with a black

148


circle, a spike pointing upward, and two spikes pointing downward. This is a typical

map indicator that shows direction for “North.” The spike to points upward is

pointing “north.” (point to the tip of the spike).

On right side, just below and to the left of the “points north” label is a label

with the word “closet.” As mentioned before, closests and bathrooms don’t have

room names. The one exception is the room with the toilet. That room’s name is

W.C., which stands for “water closet.” Water closet is a term used in England for

bathroom.

The last label is at the bottom on the right side of the mansion and contains

the word “door.” The three arrows emanating from that label point to three more

examples of doors.

The last part of the map to show you is the darkened rooms. In the map

you’re looking at, there are three darkened rooms. They are “reception,” the “small

showroom,” and the “technical design” room (Point to the three rooms). Just above

the technical design room is a dark label with the words “your task takes place in the

shaded rooms.” As already mentioned, you will be given two tasks. For each task

you will be given a map. Each map will have a different set of darkened rooms,

indicating the rooms you must visit in order to complete each task. While you are

allowed to visit other rooms, your time to complete each task is limited, so it is best

to not waste time visiting unnecessary rooms.

As a first step for each task, it is recommended that you examine the map to

determine the shortest or most efficient paths for getting from room to room, and to

return to the various rooms. As an example, in the current map, since you’re already

149


in the reception room, it would be logical to move next to the small showroom and

then the technical design room. That gives you the shortest path between the three

rooms.

Take a moment to think about the path you’d take to get from the Reception

to the Designer’s room. (Wait about 30 seconds).

To get from the Reception to the Designer’s room, you’d first move to the

Small Showroom, by going through the door on the right side of the Reception room.

Then, to move from the Small Showroom to the Designer’s Room, you’d use the

door on the right side of the Small Showroom Room. To get back to either the Small

Showroom or the Reception, you’d simply reverse your path.

Once you have a plan for how you will navigate to and from rooms, than you

would begin moving around, collecting items and attempting to open safes.

Do you have any questions?

Script for the Control Group on How to Navigate the Mansion

While the navigation map group (the treatment group) was given not only

detailed instruction on how to read the navigation map, but were also given

instruction on how to plan or find paths, the control group was given only limited

instruction on navigation. As with the navigation map group’s script, the script for

the control group included strategies for playing the game. The following script was

read to the control group.

For each of your two tasks, you will need to navigate to three rooms and

return to the rooms by retracing your path. For each task, you will be told the name

of the rooms you need to visit. As just shown, the interface includes a window that

150


displays the name of the room you’re in. Be sure to keep track of your room location.

Because you will need to find your way and than find your way back, use whatever

method you think will help to keep track of where you’ve been and the path you’ve

taken.

First Game.

After players were given instruction on navigating the environment, they

were given their first Task Completion Form (see Figure 6), which listed two of the

three rooms involved in the study and the safes they would need to open in those

rooms. Participants were told to mark off safes as they opened them. They were then

prompted to open a game already in progress. Once the game was open, they were

then told the names of the three rooms involved in the first game (Reception room,

Small Showroom, and Technical Design room) and told to take note that only two

rooms (Small Showroom and Technical Design room) and their safes were listed on

the Task Completion Form. Participants were told that they were currently in the

third room (Reception Room) and the safes for that room had already been opened

for them and the safes’ contents were in their inventory.

Those in the treatment group were then handed their navigation map for the

first game (see Figure 8). They were told that the shaded rooms for the first game

were the same three rooms that were shaded on the learning map. Both groups were

reminded not to forget to look at objects in the rooms, including the room they were

currently in, the Reception Room. Finally, participants were told they would have 15

minutes to find and open the safes and were told to begin. After fifteen minutes,

participants were prompted to save their game and exit SafeCracker. During the save

151


process, participants were given instruction on how to name their file. They were

told to use the three digit number they were randomly assigned and to add a hyphen

and a one to the end of the number. For example, if the participant’s number was

803, the filename would be 803-1. Participants were told that the next time they

saved the game they would enter their number and a hyphen followed by the number

two (e.g., 803-2).

Creating the Knowledge Map (Occasion 1).

Participants were told to start the Knowledge Mapping software. After asking

whether they had any questions, participants were told they would have seven

minutes to create a knowledge map and were told to begin. At the end of seven

minutes, participants were asked to click the X icon at the top right corner of the

screen, to exit the software. That caused a ‘save’ dialog box to open. As with the

save process for the game SafeCracker, participants were prompted to use the three

digit number they were randomly assigned and to add a hyphen and a one to the end

of the number. For example, if the participant’s number was 803, the filename would

be 803-1. Participants were told that the next time they saved the knowledge map

they would enter their number and a hyphen followed by the number two (e.g., 803-

2).

Problem Solving Strategy Questionnaire (Occasion 1).

Participants were handed the Problem solving Strategies questionnaire and

told how to fill it out. They were told the questionnaire was four pages long and

contained two questions—a retention and a transfer question. Participants were told

each question involved two pages with the first question beginning on page 1 and the

152


second question beginning on page 3. Participants were further told to start on

question 1 (the retention questions) and to not go to question 2 (the transfer

questions) until told to do so. Last, they were told they would be given two minutes

per question and were then told to begin. After two minutes, participants were

prompted to switch to the second question, the transfer question, located on page

three of the questionnaire. Participants were also told to remain on the second

question and not to return to the first question. They were also reminded to keep

writing until they were told to stop.

Second Game.

Upon collecting the Problem solving Strategies questionnaires, participants

were prompted to restart SafeCracker and to open a different game that was already

in progress. While the program was opening up, participants were handed their

second Task Completion Form (see Figure 7). They were told that one of the rooms

was a room included on the first task; the Technical Design room. They were told

that they would need to open the safes in that room even if they had opened them in

the first game. Those in the Treatment group were handed the navigation map for the

second game, which included the three darkened rooms for that game (see Figure 9).

Once SafeCracker was started and the appropriate game in progress was

opened, participants were told that the safes from the rooms in the first game that

weren’t part of the second game had been opened and their contents added to their

inventory. Participants were reminded that they would have 15 minutes for this game

and were told to begin. After 15 minutes, participants were prompted to save their

153


game, using their three digit number, along with a hyphen and the number two, as

the filename, and to exit SafeCracker.

Knowledge Map and Problem Solving Strategy Questionnaires (Occasion 2)

Participants were next prompted to restart the Knowledge Mapping software

and were given seven minutes to create their second knowledge map. At the end of

seven minutes, the participants were prompted to exit the software and to save their

file using their three digit number, a hyphen, and the number two as the filename.

Last, following the same procedures as for the first problem solving strategy

questionnaire, participants were given their second problem solving strategy

questionnaire (which was identical to the first problem solving strategy

questionnaire) and prompted to respond one question at a time. They were given a

total of four minutes for the questionnaire; two minutes per question.

Debriefing and Extra Play Time

Upon completion of the second problem solving strategies questionnaire,

participants were told the study was over. They were asked what they thought of the

game and if it was similar to games they’ve played or games they liked. If

appropriate, they were asked what types of games, and even specific games, they

liked. They were also asked if they had any questions. Finally, participants were told

they could continue playing the game if they were interested. The debriefing process

took approximately three minutes. The offer of extra play time was for collecting

data on continuing motivation. If a participant chose to continue playing, he or she

was coded as exhibiting continuing motivation; regardless of the amount of time he

or she continued to play. If a participant did not choose to continue playing, he or she

154


was coded as not exhibiting continuing motivation, even if he or she had indicated a

desire to continue playing.

Timing Chart for Pilot Study

Table 9 lists the activities encompassing the pilot study and the times

allocated with each activity, and ends with total time for the study (91 minutes) plus

the optional additional 30 minutes.

Table 9: Time Chart of the Pilot StudyActivity Time AllocationIntroduction and study paperwork 3 minutesSelf-regulation and demographic questionnaires 8 minutesIntroduction to knowledge mapping software 10 minutesIntroduction to SafeCracker for both groups and map reading and navigation for the treatment group

15 minutes

First game (3 rooms) plus task completion form 15 minutesKnowledge map creation (occasion 1) 7 minutesProblem solving strategy retention and transfer questionnaire (occasion 1)

4 minutes

Second game (3 rooms) plus task completion form 15 minutesKnowledge map creation (occasion 2) 7 minutesProblem solving strategy retention and transfer questionnaire (occasion 2)

4 minutes

Debriefing 3 minutesTOTAL 91 minutesOptional additional playing time Up to 30 minutes

Results of the Pilot Study

Overall, the instruments and procedures in the pilot study worked well. But

there was some need for modification and improvement of some of the instructions

given to participants. The first modification involved the amount of time allotted to

participants for filling out the demographic and self-regulation forms. Participants

were told they had eight minutes to fill out the forms. While both participants

completed both forms well within the eight minutes allotted, comments from one of

155


the participants indicated that unnecessary stress had been added by feeling that time

was limited. It was decided that, for the main study, participants would not be told

how much time they had, but would be prompted to finish soon, if time was running

out. In addition, because both participants in the pilot study finished well within the

eight minute time frame, it was determined that the time allotted for filling out the

demographic and self-regulation forms could be reduced from eight minutes to seven

minutes. This revision was made for the main study.

Adjustments to the knowledge mapping instruction. There were several small

problems discovered with the introduction of knowledge mapping. The first problem

was the introduction of extraneous cognitive load. Extraneous load refers to the

cognitive load imposed by unnecessary materials (Harp & Mayer, 1998; Mayer,

Heiser, & Lonn, 2001; Moreno & Mayer, 2000; Renkl & Atkinson, 2003; Schraw,

1998). In the pilot study, participants were asked to open the software. Once opened,

participants were told what knowledge mapping was. This explanation took

approximately one minute. Because the software was open, participants were

attending to the software while, at the same time, receiving information on

knowledge mapping. This imposed unnecessary, or extraneous, cognitive load. It

was decided that, for the main study, participants would be told about knowledge

mapping and then prompted to start the software.

Another important problem with the explanation of knowledge mapping was

with the examples given for types of links. Three types of knowledge map links were

described: temporal links, causal links, and simple relational links. In the pilot study,

examples were given from three randomly selected domains. It was decided for the

156


main study that all three examples should be within the same domain. To support

this, since all participants would be at, or had been at, the same southwestern

university, the domain was for the knowledge mapping instruction would be that

southwestern university, and all three link examples would be related to that

university. It was also determined that knowledge mapping instruction could be

reduced from the 10 minutes allotted for the pilot study to just eight minutes for the

main study.

A number of other small changes were made to the knowledge mapping

instructions. In particular, a strategy component was added to the main study. In the

main study, participants would be explicitly told that EVERY concept was

applicable to the game SafeCracker. The following instruction was also added to the

end of the knowledge mapping instruction for the main study:

Since every concept is applicable to SafeCracker, and therefore

should be used in your knowledge map, a recommended strategy is

to begin your knowledge map by opening the ‘add concept’ pull-

down menu and clicking on every concept. Then move the

concepts around so you can see all of them. Then switch to link

mode and start making connections.

Adjustments to the SafeCracker instructions. Several small flaws were found

with the script for the SafeCracker instruction. One example is how participants were

introduced to panning their view within the game environment. In the original script,

as part of the panning instructions, participants were told to “click the mouse in

middle of the screen and don’t let go.” While this seemed to the researcher to be an

157


obvious, explicit command, participants varied in where on the screen they clicked,

including toward the bottom or to the far right side. A participant also asked, “Which

middle? The middle of the monitor or the middle of the main window on the

interface.” For the main study, the script was changed to “click the mouse one or two

inches to the right of the phone’s hand piece and don’t let go.” That seemed to

alleviate the problem found in the pilot study.

As with the knowledge mapping instruction, it was determined that strategy

instruction needed to be added to the SafeCracker instruction. In a related

observation, it was noticed that both participants in the pilot study forgot to search

for clues. In an attempt to improve searching and search strategies, the following is

an example of instruction added to the script for the main study regarding a piece of

paper placed on a desk in the game: “Go ahead and click on it. Notice the diagrams.

These might be important for opening a safe. You might want to write them down

later, when you start playing the game.” The following search instructions,

reminders, and strategies were added to the end of the instructions for the navigation

map group in the main study.

Once you have a plan for how you will navigate to and from

rooms, then you would begin moving around, collecting items

and attempting to open safes. REMEMBER, IT IS VERY

IMPORTANT THAT YOU LOOK AT ALL THE ITEMS IN

ROOMS, TO FIND CLUES THAT MIGHT HELP OPEN

SAFES. Not everything gets added to your inventory. You may

need to write things down.

158


The following search instructions, reminders, and strategies were added to

the instructions for the control group in the main study.

Once you have a plan for how you will navigate to and from

rooms, then you would begin moving around, collecting items

and attempting to open safes. REMEMBER, IT IS VERY

IMPORTANT THAT YOU LOOK AT ALL THE ITEMS IN

ROOMS, TO FIND CLUES THAT MIGHT HELP OPEN

SAFES. Not everything gets added to your inventory. You may

need to write things down.

An important change was the addition of time added for navigation

map training for the treatment group. Originally, 15 minutes was allotted

to SafeCracker instruction. While this was sufficient time for the control

group, the treatment group needed more time. It was determined that eight

extra minutes were needed for map instructions in the main study.

Adjustments to the problem solving strategy questionnaire instructions. The

next change involved the problem solving strategy questionnaire. In the pilot study,

one participant switched to the second question before the two minutes allotted for

answering the first question were up. It was determined that, for the main study,

participants would be explicitly told “do not go to the second question until told to

do so. Continue to work on the first question for the full two minutes. And once I tell

you to go to the second question, do not return to the first question; stay on the

second question.”

159


Adjustments to the task completion form. One of the safes listed in the Task

Completion Form was the Strongbox in the Storeroom, which was connected to the

Technical Design room. This room and safe appeared on the task completion form

for both tasks (Task 1 and Task 2). While this was an accurate description of the safe

and its location, the strongbox was inside a drawer in a file cabinet. This confused

participants in the pilot study. For the main study, the text on both Task Completion

forms was changed from “Strongbox (in storeroom)” to “Strongbox (file cabinet in

storeroom). Since this safe appeared on both task completion forms, the text was

changed on both forms. See Figures 6 and 7 for the Task Completion forms used in

the pilot study and Figures 15 and 16 for the Task Completion forms used in the

main study.

In summary, running the pilot study resulted in confirming the utility of all

instruments. All instruments worked as expected, none were extraneous, and no

additional instruments were needed. However, modifications were made to the Task

Completion forms (see Figures 6 and 7 for the original forms and Figures 15 and 16

for the revised forms)) and the SafeCracker instructions (see the relevant sections

under “Pilot Study” and “Main Study” for the original and revised scripts). Changes

were also made to the study timeline (see Table 10 at the end of the descriptions of

the main study), because it was discovered that some processes could occur more

quickly, while one instruction (map instruction) needed additional time. The pilot

study timeline encompassed 91 minutes (see Table 9). The main study timeline

would encompass 96 minutes (see Table 10).

160


Procedure for the Main Study

The main study began with introducing the participants to the objective of

the experiment, describing the experiment as an examination of methods that might

help student performance when using a video game for learning, but not discussing

the issue of navigation maps. Next, participants and the researcher signed a consent

form and participants were assigned a three-digit number that had been randomly

generated prior to the study. This process took approximately three minutes.

Demographic and Self-Regulation Questionnaires.

Following the brief introduction, participants were asked to fill out the

demographic and self-regulation questionnaires (see the sections “Demographic,

Game Play, and Game Preference Questionnaire” and “Self-Regulation

Questionnaire” for descriptions of these questionnaires). Participants were given

seven minutes to fill out the two forms, but were not told there was a time limit. If

the seven minute time limit was imminent, and it appeared a participant might not

finish in time, that participant was told he or she only had a minute or two left. This

only happened once during the study (one participant) and that participant was given

an extra minute to finish the questionnaires. Most participants finished filling out the

two questionnaires within five minutes. For the Demographic, Game Play, and Game

Preference Questionnaire, if a participant asked what a game term meant, he or she

was prompted to enter a zero for their Likert-type response.

Introduction to Using the Knowledge Mapping Software.

Following administration of the demographic and self-regulation

questionnaires, participants were introduced to the knowledge mapping software.

161


This process took approximately eight minutes. Before being asked to start the

knowledge mapping software, knowledge mapping was explained to the participants,

by using their southwestern university as the domain. During the explanation, after

numerous concepts were listed, such as school, university, classroom, book, teacher,

student, sorority, fraternity, study, and party, three link examples were given. For an

example of a temporal link, the phrase “Study before tests” was given, where before

was the temporal link. For an example of a causal link, the phrase “Studying

improves grades” was given, where improves was the causal link. For a simple

relational link, the phrase “Classrooms contain book” was given, where contains was

the relational link. So that participants understood the reason for creating a

knowledge map, they were told that research has provided strong evidence that a

person’s ability to create a knowledge map is directly related to that person’s

understanding of a subject matter; that is, the more accurate and the more complete

the knowledge map, the better a person understands a domain.

Participants were then prompted to start the knowledge mapping software.

Once the software was started, the interface was explained, by describing the

function of the ‘Add Concepts’ menu item and the three on-screen buttons (see

Figure 2). Participants were asked to click on a concept to add it to the screen.

Participants were then asked to add three more concepts “for a total of four

concepts.” Participants were next prompted to open the ‘Add Concept’ pull-down

menu and to take note that the four concepts they selected were grayed out in the

menu. It was explained that a concept could only be added once, since it could have

as many links going to it or coming from it as desired. Participants were also told

162


that every concept in the pull-down menu applied to the game SafeCracker, therefore

every concept could be used in a knowledge map.

Next, participants were then prompted to move the four concepts around to

form a large box. Once completed, the three ‘Mode’ buttons near the bottom of the

screen were explained, as well as the display box to the right of the buttons. It was

explained that the reason they (the participants) were able to move the concepts

around was because they were in Move mode, as indicated by the word Move in the

display box. The participants were told that once they clicked on a mode button, they

would remain in that mode until they clicked another mode button. They were then

asked to click the Link mode button and were pointed to the display box to see that it

now showed the word Link.

Participants were told to click in the middle of one concept and drag to the

middle of another concept before letting go of the mouse, in order to generate a link.

Once they did so, and the link dialog box opened, participants were told to click on

the dialog box’s pull-down menu and look at the choices given for links, such as

contains, leads to, or requires. They were asked to pick a link and not to worry about

the appropriateness of the link. They were told that, for now, the researcher was only

concerned that they learn to use the software and didn’t care whether they created an

accurate knowledge map.

After participants successfully created the first link, they were prompted to

add at least five more links. They were told that, for the purposes of instruction, one

concept needed to have only one link, either going to it or coming from it, and that

the rest of the concepts could have as many links attached to them as desired. Once

163


enough links were added by all participants, the participants were asked to click on

the Erase mode button, to switch to Erase mode. They were prompted to look at the

display window to see that it now displayed the word Erase.

Participants were told not to click on anything until explicitly told to do so.

They were then told that the way to erase a link was to click on the words attached to

the link (such as causes or leads to). Participants were then prompted to click on the

word of the link that was connected to the concept that had only one link. That left a

concept with not links attached to it. After that, participants were told that the way to

erase a concept was to click directly on the concept. They were then prompted to

click on the concept that no longer had any links. Next, individually, the researcher

told each participant a specific concept to click on. The concept selected was

whichever concept had the largest number of links connected to it. Upon clicking the

concept, the concept and all its links were erased. Some participants were so

surprised that they made an audible sound of shock. Participants were told that the

software had no undo button, and that if they accidentally erased a concept and all its

links, they would need to recreate them. They were told that the moment they were

done erasing, they should immediately switch to either Move or Link mode, to avoid

erasing accidentally. They were then prompted to switch to Move mode.

Because participants had begun with 4 concepts and had erased two of

them, each participant now only had two concepts on the screen. In almost all cases,

there was also a link between those two remaining concepts. If there wasn’t, those

participants who didn’t have a link present were prompted to add a link, by switching

to Link mode, and then prompted to switch back to Move mode. Then participants

164


were asked to click and drag one of the concepts and were shown that, as a concept

was moved, its links moved with it. They were reminded that all the concepts in the

software applied to SafeCracker and that a recommended strategy for creating a

knowledge map was to begin by adding all the concepts to the screen and then move

the concepts around so they could begin creating links. They were told that, as the

screen got crowded, they could move concepts around and those concepts’ links

would go with them. Participants were then asked if they understood how to use the

software. Upon receiving a positive response, they were shown how to exit the

software and how to save their file when asked to do so later. See the section

“Introduction to Using the Knowledge Mapping Software” under the Pilot Study for

details on file saving.

Introduction of the Game SafeCracker.

A script was used to introduce the game SafeCracker, to ensure equivalent

learning by all participants. The script used for the main study was the results of

revisions made to the script for the pilot study, based on observations, discussions

and participants comments that occurred during and after the pilot study. The

procedures for teaching SafeCracker were the same as those used in the pilot study,

with the addition of reminders to all participants to check for clues as they played the

game and that not all clues were added to the inventory, so they might need to write

some things down on the scratch paper they were provided. Participants were also

told they could get as many pieces of scratch paper as desired. Fifteen minutes were

allotted to teaching SafeCracker. The following is the script used for the main study.

165


SafeCracker training script. In this study, you will be asked to accomplish a

series of tasks. The tasks will be to locate and open various safes in various rooms in

a mansion. In order to open some of these safes, you will need to find certain items.

You will be told which rooms to visit. Those rooms contain all the items needed and

all the safes you will need to open. You do not need to spend time in any other

rooms. Even though the mansion has two floors, all the rooms you will visit are on

the first floor. Do not go to the second floor.

Your goal is to open all the safes in the rooms you are given. For each room,

you will be told the room’s name, for example, the Small Showroom. Together, we

will walk through the steps needed to find and enter the mansion. Then, we will walk

through searching the first room and opening one safe. After that, you will be given

the names of several rooms and will be required to find the rooms and open the safes.

From this point on, DON’T DO ANYTHING UNLESS I EXPLICITLY TELL YOU

TO with phrases like “OKAY, DO IT NOW” or “GO AHEAD AND DO IT.” Does

everyone understand?

(GETTING INTO THE MANSION.) Once I tell you to click the “new”

button to begin a new game, you’ll see the game’s main interface screen and a

phone. You’ll be in a phone booth facing the phone. The phone will be ringing, but

you won’t hear it because the sound is turned off on your computer. As you move

your cursor to the phone’s hand piece, a hand symbol will appear on your cursor.

That means you can click on the phone piece to remove it from its hook. Anytime

you see a hand symbol it means you can click on something. Once you remove the

hand piece, a voice will begin speaking. Unfortunately, you won’t be able to hear it

166


because, as already mentioned, the sound is turned off on your computer. So right

now, I’m going to tell you the most important information you would have heard.

And I need you to write it down on your scratch paper as I say it to you. That

information is a four-digit code you’re going to need to use in order to enter the

mansion. That code is 1923. Write that down now; 1923.

Once you remove the hand piece, wait about five seconds, then move your

cursor back to the hook. When you see the hand symbol, click to hang up the phone,

then wait for further instructions. To repeat, you will click the phone piece, wait

about five seconds, and click the handle to hang up the phone. After that, you will do

nothing. That includes not moving the cursor. Do you have any questions? Okay, go

ahead now and click NEW to start the game.

(Once everyone’s done listening to the message). Move your cursor about

two inches to the right of the hand piece (wait), hold down the cursor, and move your

mouse left and right. Your view will pan left and right. The further you move left or

right, the faster the scene will pan. You can also move the cursor up or down to tilt

up or down a little. If you stop moving the mouse but continue to hold the mouse

button down, the scene will continue to pan. (long pause.) You can also use the left,

right, up, and down cursor arrows pan or tilt your view.

Now let’s exit the phone booth. Rotate until you see the sidewalk. When you

let go of the cursor, you should see the hand symbol. If you don’t see the hand

symbol, rotate 180 degrees and look at the sidewalk going the other direction. Then

click once to open the phone booth door and a second time to exit the phone booth.

Only click twice. After that, wait for instructions. Okay, go ahead and do that now.

167


(Once everyone’s outside the phone booth). Now, I’m going to give you

some instructions and it’s very important that you do absolutely nothing until I tell

you to (beat). Across the street is a lit up, two story mansion. Notice how much of

your sidewalk you can currently see in front of you? When I tell you to, you’re going

to slowly rotate to the right. You’ll stop when you can see as much of the mansion as

possible, while still seeing as much of your sidewalk as you currently see. Go ahead

and do that, then wait for further instructions.

Once again, I’m going to give you a series of instructions and it’s very

important that you do not do anything until I explicitly tell you to (beat.). Do you see

those white stripes crossing the street? That’s a crosswalk. You’re going to walk

down your sidewalk, turn right and cross the street. Then you’ll turn left and walk

further down the other sidewalk. Then you’ll turn to the right and face the mansion

(beat). More specifically, you’ll take two steps down your sidewalk. You’ll turn to

the right and take two steps to cross the street. You’ll turn to the left and take two

more steps to walk further down the other sidewalk. Then you’ll turn right and face

the mansion. Okay, go ahead and get to the mansion.

(Once everyone’s facing the gate). You should be facing a gate, with the

mansion in the background. See the lock on the gate. Go ahead and click on it.

You’ll need to see the hand symbol in order to click (wait). Now, notice the three

tumblers on the lock. Go ahead and click on the tumblers, and you’ll notice that all

three can be rotated. If you rotate them to the correct pattern, the lock will open. I’ll

give you one minute to try. Go ahead now and try to open the lock.

168


(After one minute). If you haven’t opened the lock yet, set the three tumblers

to music symbols and the lock will open. Go ahead. (Once all locks are opened) Now

go ahead and move to the front door. You’ll need to navigate a little bit around that

fountain that’s up ahead. (Once everyone’s at the front door). Notice how the front

entrance has two doors. I want you do rotate so that you see the area just to the left of

the left door. Go ahead and start rotating. Notice that small gray box? Go ahead and

click on it (long pause). Now, using your mouse, click the four digits I had you write

down earlier, then click on the ENTER button on that keypad. Go ahead and do that

now. Remember to use the mouse to click on numbers, rather than using your

keyboard. (Once everyone’s gained access). Now click once to open the first door.

Now click again to open the second door and a third time to enter the mansion.

(Once everyone’s inside the mansion). Rotate left and right and you’ll notice

you’re in front of a reception desk. The secretary’s chair to your left is facing a

computer. Rotate and you should be able to see the back of the monitor. Go ahead

and navigate around until you’re facing that computer (long pause). Once you’re

there, click on the computer screen once and then wait. Go ahead. (Once everyone’s

at the computer). Click once more on the computer screen. Then wait about five

seconds.

(Once everyone is looking at the game minesweeper on the screen). Now try

to move your mouse to look around. Notice how nothing happens. This is because

you’re ‘locked’ on to the computer screen. Whenever you’re ‘locked’ onto

something, you can’t do anything else until you BACK away from that object. To do

that, click the BACK button that’s on the right side of the screen. Go ahead and do

169


that. (Once everyone’s backed up). Now, click on the blue cup to the left of the

computer. A larger view of the cup appears in a window at the bottom of the screen.

You can grab the cup on that window and rotate it. Go ahead and try that. (After

everyone’s rotated the cup).

To the left of the blue cup is a piece of paper. Go ahead and click on it.

Notice the diagrams. These might be important for opening a safe. You might want

to write them down later, when you start playing the game. If you move your cursor

near the bottom of the paper, a down arrow cursor appears, indicating that you can

click to see more of the paper. Go ahead and do that. You can also click to the right

to see more of the paper. And click near the top to see the top portion of the paper.

(Once everyone’s seen all parts of the paper). Go ahead and click your back

button (long pause). Now rotate to the right and find the next piece of paper. Go

ahead and click on that. You can move your cursor to the bottom and click to see

more. With this paper, you can’t click to see the right portion (wait). Go ahead and

click the back button. Rotate a little more to the right and you’ll see a third piece of

paper. Go ahead and click on that paper. You can click down on the paper and to the

right to see more. This paper is filled with diagrams that might be helpful for

opening safes (beat.). Once you’ve seen the whole paper, click the back button and

then don’t do anything else. (Wait until everyone’s seen both papers).

I don’t want you to click on anything else, but notice there are several books.

Some contain potentially useful information. If you rotate around, you’d see other

items around the desk that might be worth looking at, including more papers and

more books. Go ahead and rotate around. (wait a few seconds)

170


Now I’d like you to face that computer screen you went to earlier (long

pause). See that blue safe in the background? I want you to navigate to it. Go ahead

(beat). When you get there, click on the safe to lock onto it and don’t do anything

else. BE SURE NOT TO CLICK ON ANYTHING (long pause). You’ll know you’re

locked onto the safe when the button on the right side of the screen says BACK.

(Once everyone is near the safe). Once again, wait until I tell you to before

clicking on anything (beat). The safe has three red lights, three white dials, and a

handle just below the middle dial. The safe will open when the three dials are set to

the correct numbers (beat). The red lights are connected to the dials. The left dial

controls the top red light. The middle dial controls the middle red light. And the right

dial controls the bottom red light.

Go ahead and click the handle now and you’ll notice one of the lights

remains a solid green while the other two are flashing green. If a light remains green,

it means that its dial is set correctly. You can keep clicking on the handle if you want

(pause). When you click, the middle light stays solid green, so the middle dial must

be set correctly. I’ll want you to adjust the dials until all the lights stay green. Go

ahead and do that now. And remember, the middle dial is correct so don’t move it.

(Once everyone’s opened the safe). Click on the piece of paper. Click it again

to make it go away. Notice it’s been added to your inventory on the bottom right

window of the screen. Click on it in the inventory to open it up again. Click on it

once more to make it go away (beat). Now, click on the coins to add those to your

inventory as well. Some items can be used to open safes; for example, keys might be

used to open a lock or coins might be inserted into a coin slot (long pause). To use an

171


item, you click on the item to make it active. Right now, your coins are active.

Notice your inventory text? And notice that window to the left of the inventory that

shows a 3D image of the coins? Now, notice the vertical button between them that

says “USE.” That’s the button you click in order to use something. Go ahead and

click it now (long pause). If you were at something that could use the coins, they

would have been used. But since you weren’t, you might be able to use them later

(beat). Now, back away from the safe.

(Wait until everyone’s backed away from the safe). Now click on the safe to

try to lock onto it. Notice you can’t and a red error message appears at the top of the

screen. That message states that the safe has already been cracked. ONCE YOU

CLOSE A SAFE, YOU CAN NEVER OPEN IT AGAIN. SO BE SURE TO

COLLECT ALL ITEMS FROM A SAFE BEFORE BACKING AWAY.

Notice that area where the red warning sign appeared? It now says

“Reception.” That’s the name of the room you’re in (beat). When it’s not displaying

an error message, that’s the room indicator window and it lets you know the name of

the room you’re in (beat). Also notice the compass at the bottom right of the

computer screen. You might find that compass helpful for determining which

direction you’re facing or moving.

(FOR MAP USERS, READ THE “SCRIPT FOR INTRODUCING MAP

TO PARTICPANTS”)

(FOR NON-MAP USERS, READ THE “SCRIPT FOR THE CONTROL

GROUP ON HOW TO NAVIGATE THE MANSION”)

172


Introduction to Using the Navigation Map.

Those in the navigation map group (the treatment group) were next

introduced to reading the navigation map and were given instruction on path finding

and path planning. They also received instructions on strategies for playing the

game. To ensure equivalent learning by all those in the treatment group, a script was

utilized (see below). Those in the control group were given simple guidelines for

navigation. See the next section entitled “Script for the Control Group on How to

Navigate the Mansion” for information on the script given to the control group.

The script for the main study on how to use the navigation map was revised

after the pilot study, based on observations by the researcher and comments from the

participants. To support both the original and revised versions of the script, a special

version of the navigation map, a training map, was created (see Figure 14),

displaying a portion of the floor plan, along with shaded portions, labels, and arrows,

as aids to the script.

The training map was handed to each participant on a standard sheet of white

paper and contained the title “How to read the map.” As participants viewed the

training map, the script was read. In addition to training on map reading, path

planning, and path finding, the script included some strategies for playing the game.

The script also contains two words or phrases in parentheses: beat and long pause.

The term beat is a common term in script writing and refers to a “momentary pause

in dialog or action” (Armer, 1988, p. 260). The term long pause does not have a

history in script writing and is used here to indicate a pause of at least two seconds.

The following script was read to the navigation map group.

173


Training map script. This is a map of a portion of the first floor of the

mansion. You will use a map similar to this to help navigate to the various rooms.

Currently, you’re in the reception room, the large room near the middle of the

bottom portion of the map. The map shows some of the rooms on the first floor with

their related names. These will match the names of the rooms that contain the safes

you will be asked to open and the items that will help you to open the safes. The

names also match the names that appear in the name indicator on your interface,

which you have already been shown. You will not need to visit any other rooms,

unless they are along a path you take in order to get to a required room. In addition to

the room names, the map also shows the location of the doors in each room. If you

need to, you are allowed to write on this map.

This map shows a portion of the bottom floor and includes text labels

describing the most important map features. On the left side are four labels. The top

label on the left contains the words “room name” and points to the name of the room

entitled “Big Showroom.” Take a look and you’ll see that every room has a name.

Those areas that do not have names are either closets or bathrooms. For each of your

two tasks, you will be told the names of the rooms you must visit. As shown to you

earlier, there is a room name indicator on the interface. You will use this indicator to

verify which room you are in.

The middle label on the left contains the word “stairs” and points to a block

of black and gray stripes. That pattern indicates stairs. Notice that there’s another set

of stairs just to the right and a small set of stairs connecting the two.

174


On the left side, near the bottom, is a label with the word “door.” Gaps or

open spaces between rooms indicate doors. Every room has at least one door and

most have several doors.

On the left side, at the bottom, is a label with the words “Main Entrance” and

an arrow pointing to the door you came through to enter the mansion. This is the

only door on the map that is not indicated using an opening or gap. Once you began

moving around the reception room that door locked and cannot be opened. That’s

why it is not indicated by an opening.

In the middle of the map is a label with the word “toilet.” The arrow points to

a small circle, which is the symbol for a toilet. There are other bathrooms in the

mansion that have toilets, but for some reason, the people who created this map

chose to only show this toilet.

On the right side of the map are three labels. The one on the far right side of

the map and containing the words “points north” points to a symbol with a black

circle, a spike pointing upward, and two spikes pointing downward. This is a typical

map indicator that shows the direction for “North.” The spike that points upward is

pointing “north.”

On the right side, just below and to the left of the “points north” label is a

label with the word “closet.” As mentioned before, closests and bathrooms don’t

have room names. The one exception is the room with the toilet. That room’s name

is W.C., which stands for “water closet.” Water closet is a term used in England for

bathroom.

175


The last label is at the bottom on the right side of the mansion and contains

the word “door.” The three arrows emanating from that label point to three more

examples of doors.

The last part of the map to show you is the darkened rooms. In the map

you’re looking at, there are three darkened rooms. They are “reception,” the “small

showroom,” and the “technical design” room. Just above the technical design room

is a dark label with the words “your task takes place in the shaded rooms.” As

already mentioned, you will be given two tasks. For each task, you will be given a

map. Each map will have a different set of darkened rooms, indicating the rooms you

must visit in order to complete each task. While you are allowed to visit other rooms,

your time to complete each task is limited, so it is best to not waste time visiting

unnecessary rooms.

As a first step for each task, it is recommended that you examine the map to

determine the shortest or most efficient paths for getting from room to room, and

return to the various rooms. As an example, in the current map, since you’re already

in the reception room, if you wanted to go to the “Small Showroom,” you’d use the

right door of the reception room to enter the “Small Showroom.” If you wanted to go

from the small showroom to the technical design room, there are no doors leading

directly from one room to the other; there are no openings. Instead, you’d need to go

first to your right and enter the “Designer’s room.” Then you move up the left side of

that room and through a door that leads into the “Technical Design room.” To return

to the “Small Showroom,” you’d simply reverse your path.

176


Once you have a plan for how you will navigate to and from rooms, then you

would begin moving around, collecting items and attempting to open safes (Pause).

Remember, it is very important that you look at all the items in rooms, to find clues

that might help open safes (beat). Not everything gets added to your inventory. You

may need to write things down (long pause). Do you have any questions?

Script for the Control Group on How to Navigate the Mansion

While the navigation map group (the treatment group) was given not only

detailed instruction on how to read the navigation map, but were also given

instruction on how to plan or find paths, the control group was given only limited

instruction on navigation. As with the navigation map group’s script, the script for

the control group included strategies for playing the game. The following script was

read to the control group.

For each of your two tasks, you will need to navigate to three rooms and

return to the rooms by retracing your path. For each task, you will be told the name

of the rooms you need to visit. As just shown, the interface includes a window that

displays the name of the room you’re in. Be sure to keep track of your room location.

Because you will need to find your way and than find your way back, use whatever

method you think will help to keep track of where you’ve been and the path you’ve

taken (beat). Note: You will need to go through other non-task related rooms to get

to your rooms. And remember, it is very important to look at all the items in rooms,

to find clues that might help open safes (beat). Not everything gets added to your

inventory; you may need to write things down (long pause). Do you have any

questions.

177


While eight extra minutes were needed for training the navigation map group

on use of the navigation map, only one or two minutes were needed for providing

navigation guidance to the control group. Therefore, the control group’s total

participation time was approximately 6 minutes less than the navigation map group’s

total participation time (i.e., 90 minutes versus 96 minutes, respectively).

First Game.

As with the pilot study, after participants were given instruction on

navigating the environment, they played their first game. This phase of the study

began with handing participants their first Task Completion Form (Figure 15). For a

complete listing of instructions given for the task completion form, see the “First

Game” section under the topic “Pilot Study.”

Figure 15: Task Completion Form 1 for Main Study

178


Those in the treatment group were then handed their navigation map for the

first game (see Figure 8). As with the pilot study, they were told that the shaded

rooms for the first game were the same three rooms that were shaded on the learning

map. Both groups were reminded not to forget to look at objects in the rooms,

including the room they were currently in, the Reception Room. Finally, participants

were told they would have 15 minutes to find and open the safes and were told to

begin. After 15 minutes, participants were prompted to save their games and exit the

software using the same procedures as used in the pilot study.

Creating the Knowledge Map (Occasion 1)

Participants were next prompted to start the Knowledge Mapping software.

After asking whether they had any questions, participants were told they would have

seven minutes to create a knowledge map and told to begin. After seven minutes,

participants were prompted to save their files and exit the software using the same

procedures as in the pilot study (see the section “Creating the Knowledge Map

(Occasion 1)” under the topic “Pilot Study”).

Problem Solving Strategy Questionnaire (Occasion 1)

As with the pilot study, participants were given the first problem solving

strategy questionnaire for the first game and told how to fill it out. They were told to

stay on the first question until told to go to the second question and that once they

were on the second question they were to remain there and not go back to the first

question. Participants were given four minutes for the questionnaire, at two minutes

per question.

179


Second Game

Procedures for the second game were the same as for the second

game of the pilot study. They were also handed the Task Completion form for the

second task (Figure 16). This form had been modified from the Task Completion

form used in the pilot study (see Figure 7). The wording for the Strongbox safe in the

Technical Design room was changed from “Strongbox (in storeroom)” to “Strongbox

(file cabinet in storeroom).” Those in the navigation map group were handed the

navigation map for the second game (see Figure 9), which included the darkened

rooms of the game. Those in the navigation map group were reminded that the

darkened rooms represented the rooms that contained the safes they would need to

locate and open.

Figure 16: Task Completion Form 2 for Main Study

180


All participants were told that they would begin the second game in the

Technical Design room, which was one of the three rooms included in the first game.

Participants were told they would need to open the safes in the Technical Design

room, even if they had already opened those safes in the first game. They were also

told that the safes from the other two rooms from the first game had already been

opened for them and the contents of those safes were in their inventory. Participants

were reminded that even though the safes from the other two rooms had been

opened, they might still want to revisit those rooms to look for clues.

Participants were told they would have 15 minutes for this game and were

prompted to begin. After 15 minutes were up, participants were asked to save their

game and exit the software using the same procedures given in the pilot study (see

the section “Second Game” under the topic “Pilot Study”).

Knowledge Map and Problem Solving Strategy Questionnaires (Occasion 2)

Participants were next prompted to restart the Knowledge Mapping software

and were given seven minutes to create their second knowledge map. After seven

minutes, participants were asked to save their files and exit the software using the

same procedures used in the pilot study (see the section “Knowledge Map and

Problem Solving Strategy Questionnaires (Occasion 2)” under the topic “Pilot

Study”). Last, participants were given their second Problem Solving Strategies

Questionnaire, which was identical to the first Problem Solving Strategies

Questionnaire and prompted to respond one question at a time, as with the prior

questionnaire. They were given a total of four minutes for the questionnaire; two

minutes per question.

181


Debriefing and Extra Play Time

Upon completion of the second Problem Solving Strategies Questionnaire,

participants were told the study was over. They were asked what they thought of the

game and if it was similar to games they’d played or games they liked. If

appropriate, participants were asked what types of games, and even specific games,

they liked. They were also asked if they had any questions. Finally, participants were

told they could continue playing the game for up to 30 minutes if they were

interested. Debriefing took approximately three minutes. The offer of extra play time

was for collecting data on continuing motivation. If a participant chose to continue

playing, he or she was coded as exhibiting continuing motivation; regardless of the

amount of time he or she continued to play. If a participant did not choose to

continue playing, he or she was coded as not exhibiting continuing motivation, even

if he or she had indicated a desire to continue playing.

Timing Chart for Main Study

Table 10 lists the activities encompassing the main study and the times

allocated with each activity, and ending with total time (96 minutes) and the

optional 30 minutes of playing time. The Pilot Study activity “Introduction to

SafeCracker and Map Reading” which encompassed 15 minutes was divided into

two activities for the main study: one activity was “Introduction to SafeCracker”

which encompassed 15 minutes. The second activity, immediately following the

“Introduction to SafeCracker,” was “Introduction to map reading for the treatment

group” and encompassed an additional 8 minutes. While those in the control group

did not receive map reading instructions, they did receive some navigational

182


instruction during this time, which required approximately two minutes. Therefore,

the amount of time required for the control group to complete the study was

approximately 6 minutes less than the time required for the treatment group to

complete the study: 90 minutes versus 96 minutes.

Table 10: Time Chart of the Main StudyActivity Time AllocationIntroduction and study paperwork 3 minutesSelf-regulation and demographic questionnaires 7 minutesIntroduction to knowledge mapping software 8 minutesIntroduction to SafeCracker 15 minutesIntroduction to map reading for the treatment group 8 minutesFirst game (3 rooms) plus task completion form 15 minutesKnowledge map creation (occasion 1) 7 minutesProblem solving strategy retention and transfer questionnaire (occasion 1)

4 minutes

Second game (3 rooms) plus task completion form 15 minutesKnowledge map creation (occasion 2) 7 minutesProblem solving strategy retention and transfer questionnaire (occasion 2)

4 minutes

Debriefing 3 minutesTOTAL 96 minutesOptional additional playing time Up to 30 minutes

183


CHAPTER 4

ANALYSIS AND RESULTS

Descriptive and inferential statistical results based on data collected from the

main study are presented. SPSS 13.0 for Windows program (2002) was used to

analyze the data. The data were analyzed to assess the study’s hypotheses.

Research Hypotheses













better performance.




group).

184


Content Understanding Measurement

Content understanding was assessed through construction of knowledge

maps, created on two occasions; occasion 1 and occasion 2. Occasion 1 was after the

first game, and occasion 2 was after the second game. Table 11 shows the mean

scores and standard deviations for knowledge map creation for the control group,

navigation map group (the treatment group), and the both groups combined (total)

for two occasions (occasion 1 and occasion 2) and the amount of improvement for

each group from occasion 1 to occasion 2. Improvement is defined as a group’s

occasion 2 score minus the group’s occasion 1 score.

Table 11Descriptive Statistics of Knowledge Map Occasion 1 and Occasion 2 Scores for the Control Group, Navigation Map Group, and Both Groups Combined

Group Mean SDControl (n = 31)

Occasion 1 6.57 2.85Occasion 2 7.55 3.37Improvement 1.00 1.17

Navigation Map (n = 33)Occasion 1 6.51 2.30Occasion 2 7.85 3.06Improvement 1.34 2.75

Total (n = 64)Occasion 1 6.54 2.56Occasion 2 7.70 3.19Improvement 1.16 2.68

For the control group, the mean scores for knowledge map construction

occasion 1 and occasion 2 were 6.57 and 7.55, respectively. For the navigation map

group, the mean scores for knowledge map construction occasion 1 and occasion 2

were 6.51 and 7.85, respectively. For both groups combined, the mean scores for

knowledge map construction occasion 1 and occasion 2 were 6.54 and 7.70,

185


respectively. The mean scores for improvement in knowledge map construction from

occasion 1 to occasion 2 for the control group, the navigation map group, and both

groups combined were 1.00, 1.34, and 1.16, respectively.

There was no significant difference in the improvement of content

understanding between the control group and the navigation map group, t(62) = .54,

p = .59, Cohen’s d effect size index = .17. Cohen (1988) defined d as the difference

between the means divided by the standard deviation of each group and also defined

d = .2 as small, d = .5 as medium, and d = .8 as large effect sizes. The effect size

of .17 for the differences in improvement in knowledge mapping scores between the

control and navigation groups was below the smallest effect size, indicating a

negligible effect.

Another way to look at the knowledge map data is to calculate the percentage

of the mean scores for knowledge mapping by the control group, the treatment

group, and both groups combined compared to the mean scores of the three expert

maps. The scores for the expert maps were 90, 99, and 54. The mean score for the

expert maps was 81. The percentage of a group’s mean score is calculated by

dividing the group’s mean score by the experts’ mean score. For example, the mean

score for the control group for occasion 1 was 6.57 (see Table 11). Dividing that

score by the expert mean score (6.57 divided by 81) yielded a mean percentage of

8.11% for the control group for occasion 1. That is, the control group’s mean score

represents 8.11% of the mean score achieved by the experts. As seen in Table 12,

mean percentages for the control group were 8.11% for the occasion 1 and 9.32% for

occasion 2. Mean percentages for the navigation map group were 8.04% for occasion

186


1 and 9.69% for occasion 2. Mean percentages for both groups combined were

8.07% for occasion 1 and 9.51% for occasion 2.

Table 12Descriptive Statistics of the Percentage of Knowledge Map Occasion 1 and Occasion 2 Scores for the Control Group, Navigation Map Group, and Both Groups Combined

Group Mean SD

Control (n = 31)Occasion 1 8.11% 3.52%Occasion 2 9.32% 4.16%Improvement 1.23% 1.44%

Navigation Map (n = 33)Occasion 1 8.04% 2.84%Occasion 2 9.69% 3.78%Improvement 1.65% 3.40%

Total (n = 64)Occasion 1 8.07% 3.16%Occasion 2 9.51% 3.94%Improvement 1.43% 3.31%

Improvement percentages for the control group, the navigation map group,

and both groups combined were 1.23%, 1.65%, and 1.43%, respectively. There was

no significant difference in the improvement of content understanding between the

control group and the navigation map group, t(62) = .54, p = .59, Cohen’s d effect

size index = .17. The effect size of .17 for the differences in improvement in

knowledge mapping scores between the control and navigation groups was below the

smallest effect size, indicating a negligible effect.

The occasion 1 and occasion 2 knowledge map scores for the control group

were significantly correlated, r = .65, p < .01, as were those of the navigation map

group, r = .50, p < .01. A t-test also confirmed that no significant difference was

found between the occasion 1 scores of the two groups, t(62) = -.10, p = .92. A t-test

187


also confirmed there was no significant difference between the occasion 2 scores of

the two groups, t(62) = .37, p = .71.

A mixed-groups, repeated measures factorial ANOVA was also performed to

examine the effects of the use or non-use of a navigation map on content

understanding as exhibited through knowledge map construction on occasion 1 and

occasion 2. Table 13 shows the means for the conditions of the design. There was no

interaction between treatment and occasion F(1,62) = .29, p = .59. There was also no

main effect for group, F(1,62) = .03, p = .86. There was a main effect of occasion,

F(1,62) = 11.84, p = .001, with higher knowledge mapping scores in occasion 2 than

in occasion 1, for both the control group and the navigation map group.

Table 13Knowledge Mapping Means by Group by OccasionGroup KM Occasion 1 KM Occasion 2

Control (n = 31) 6.57 (2.85) 7.55 (3.37) 7.06 (3.11)Treatment (n = 33) 6.51 (2.30) 7.85 (3.06) 7.18 (2.68)

6.54 (2.58) 7.70 (3.22)

Interrater Reliability of the Problem Solving Strategy Measure

Two researchers independently assigned an expert idea unit to each of the

1520 participant problem solving strategy retention and transfer responses, assigning

expert retention idea units to the participants’ retention responses and expert transfer

idea units to the participant’s transfer responses. Then, the two expert lists were

compared. The two experts had agreed on 1275 of the 1520 idea units, which was

83.9% agreement (1275/1520 = .839).

The percentage of interrater agreement was then analyzed by problem solving

strategy retention responses and by problem solving strategy transfer responses. Of

188


the 1520 participant responses, 1021 were problem solving strategy retention

responses and 499 were problem solving strategy transfer responses. Table 14 shows

the number of responses for each of the 28 problem solving strategy retention idea

units by each of the two raters (see Table 7 for a description of the retention idea

units). Note that Table 14 indicates there were 29 idea units, plus an idea unit

numbered as zero. The 29th idea was an error entered by one of the experts during

coding. The zero indicates a participant response that did not fit into any idea unit.

There were 161 problem solving strategy retention responses coded with zero. For

example, on a number of occasions, participants responded with a single word, such

as clue or hint. Because an idea unit required a verb and a noun, single-word

response could rarely be matched with an idea unit. Those responses were coded as

zero. A few single word responses did receive matches. For example, map was

interpreted as use map and compass was interpreted as use compass, since that is the

only logical interpretation of the word within the context of SafeCracker. But since a

word like clue might mean use clue, find clue, interpret clue, search for clue, etc.,

the experts were unable to determine an appropriate match.

Table 15 shows the number of responses for each of the 21 problem solving

strategy transfer idea units by each of the two raters (see Table 8 for a description of

the transfer idea units). Note, no participant response matched expert idea unit

number 21, therefore, that number does not appear in the chart. Similar to the

problem solving strategy retention responses, 134 of the participant responses to the

problem solving strategy transfer question did not match an expert idea unit and were

given a value of zero. For example, on a number of occasions, participants simply

189


responded that a particular game feature was difficult, but did not indicate whether it

needed modification. Those responses were coded as zero.

Table 14Matrix of the Number of Participant Responses Assigned to Each Idea Unit in the Problem Solving Retention Measure Based on Two Rater’s Scoring

Expert 10 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Expert

2

0 161 3 5 6 1 3 1 1 21 5 12 1 12 4 1 37 3 3 1 13 3 8 178 3 14 1 6 15 1 34 16 6 24 17 1 218 1 1 66 1 19 3 3 1 1 6410 3 311 12 1 12 3 1 1112 1913 1 3714 1 415 2 116 117 218 1 2 419 2 1 220 1 12122 1 123 224 2252627 128 1 1

Total 203 30 45 203 15 45 31 23 68 79 3 14 19 38 5

190


Table 14 (Continued)Matrix of the Number of Participant Responses Assigned to Each Idea Unit in the Problem Solving Retention Measure Based on the Two Rater’s Scoring

Expert 1 Total15 16 17 18 19 20 22 23 24 26 27 28 29

Expert

2

0 3 5 2 3 2 1 1991 192 1 1 523 3 1 1974 85 366 1 1 337 228 1 719 7210 1 711 4012 1913 1 3914 515 6 1 1 1116 23 2417 11 1 1 1518 42 4919 7 1 1 1420 1 321 1 122 3 1 2 823 3 15 2 1 2324 4 1 1 825 1 126 2 23 2527 1 2 428 4 10 16

Total 6 24 23 50 9 7 6 23 2 25 13 11 1 1021

191


Table 15Matrix of the Number of Participant Responses Assigned to Each Idea Unit in the Problem Solving Transfer Measure Based on the Two Rater’s Scoring

Expert 1 Total 0 1 2 3 4 5 6 7 8 10 11 12 13 14 15 16 17 18 19 20

Expert

2

0 134 1341 1 12 54 543 18 184 2 25 2 26 43 437 13 138 5 510 15 1511 3 3

12 101 10113 40 4014 30 3015 9 916 6 617 4 418 12 1219 4 420 3 3

Total 134 1 54 18 2 2 43 13 5 15 3 101 40 30 9 6 4 12 4 3 499

As can be see in Table 14, the number of idea units agreed upon for the

problem solving strategy retention responses was 823. Agreement is represented in

Table 14 by the diagonal listing of numbers (i.e., 161, 12, 37, 178 … 1, 23, 2, and

10). With a total of 1021 problem solving strategy retention responses, the two raters

agreed on 80.6% of the responses (823/1021 = .806). As can be seen in Table 15, the

number of idea units agreed upon for the problem solving strategy transfer responses

was 355. Agreement is represented in Table 15 by the diagonal listing of numbers

(i.e., 134, 1, 54, 18, 2 … 4, 12, 4, and 3). With a total of 499 problem solving

192


strategy transfer responses, the two raters agreed on 90.6% of the responses (452/499

= .906). All 352 disagreements for the problem solving strategy retention and

transfer responses combined were resolved through discussions by the two raters,

ending the rating process with 100% agreement on all retention and transfer

responses. The data used for reporting the results of the problem solving strategy

measure represent the data after all rater disagreements were resolved; that is, after

100% agreement had been reached.

Problem Solving Strategy Measure

Problem solving strategy retention and transfer were assessed through a

problem solving strategy questionnaire which contained two questions: a retention

question a transfer question (see Table 6). The questionnaire was administered on

two occasions, occasion 1 and occasion 2. Occasion 1 was after the first game, and

occasion 2 was after the second game.

Retention question. As shown in Table 16, for the control group, the mean

scores for the problem solving strategy retention occasion 1 and occasion 2

responses were 6.19 and 5.71, respectively. For the navigation map group, the mean


responses were 6.06 and 5.55, respectively. For both groups combined, the mean


responses were 6.13 and 5.63, respectively. Mean scores for improvement from

occasion 1 to occasion 2 for the control group, the navigation map group, and both

groups combined were -.48, -.52, and -.50, respectively.

193


Table 16Descriptive Statistics of Problem Solving Strategy Retention Occasion 1 and Occasion 2 Scores for the Control Group, Navigation Map Group, and Both Groups Combined


Occasion 1 6.19 3.48Occasion 2 5.71 2.76Improvement -.48 2.87

Navigation Map (n = 33)Occasion 1 6.06 2.30Occasion 2 5.55 3.15Improvement -.52 2.25

Total (n = 64)Occasion 1 6.13 2.91Occasion 2 5.63 2.95Improvement -.50 2.55

There was no significant difference in the improvement of problem solving

strategy retention between the control group and the navigation map group, t(62) =

-.05, p = .96, Cohen’s d effect size index = .05. The effect size of .05 for the

differences in improvement between the control and navigation groups indicated a

negligible effect.

The occasion 1 and occasion 2 problem solving strategy retention scores for

the control group were significantly correlated, r = .60, p < .01, as were those of the

navigation map group, r = .70, p < .01. A t-test also confirmed that no significant

difference was found between the occasion 1 scores of the two groups, t(62) = -.18, p

= .86. A t-test also confirmed there was no significant difference between the

occasion 2 scores of the two groups, t(62) = -.22, p = .83.

Another way to look at the problem solving strategy retention data is to

calculate the percentage of the mean scores for problem solving strategy retention by

194


the control group, the treatment group, and both groups combined compared to the

number of expert idea units created for the problem solving strategy retention

question. The experts defined 28 idea units related to the problem solving strategy

retention question (see Table 6). The percentage of a group’s means score is

calculated by dividing the group’s mean score by 28—the number of expert problem

solving strategy retention idea units. For example, the mean score for the control

group for occasion 1 was 6.19 (see Table 16). Dividing that score by the number of

expert idea units (6.19 divided by 28) yielded a mean percentage of 22.11% for the

control group for occasion 1. That is, the control group’s mean score for the total

number of problem solving strategy retention idea units generated is equal to 8.11%

of the total number of expert problem solving strategy retention idea units.

As seen in Table 17, mean percentages for the control group were 22.11% for

the occasion 1 and 20.39% for occasion 2. Mean percentages for the navigation map

group were 21.64% for occasion 1 and 19.82% for occasion 2. Mean percentages for

both groups combined were 21.89% for occasion 1 and 20.11% for occasion 2. There

was no significant difference in the improvement of problem solving strategy

retention between the control group and the navigation map group, t(62) = -.05, p

= .96, Cohen’s d effect size index = .05. The effect size of .05 for the differences in

improvement between the control and navigation groups indicated a negligible

effect.

195


Table 17Descriptive Statistics of the Percentage of Problem Solving Strategy Retention Occasion 1 and Occasion 2 Scores for the Control Group, Navigation Map Group, and Both Groups Combined


Occasion 1 22.11% 12.43%Occasion 2 20.39% 9.86%Improvement -1.71% 10.25%

Navigation Map (n = 33)Occasion 1 21.64% 8.21%Occasion 2 19.82% 11.25%Improvement -1.86% 8.04%

Total (n = 64)Occasion 1 21.89% 10.39%Occasion 2 20.11% 10.54%Improvement -1.79% 9.11%

The occasion 1 and occasion 2 problem solving strategy retention scores for

the control group were significantly correlated, r = .60, p < .01, as were those of the

navigation map group, r = .70, p < .01. A t-test also confirmed that no significant

difference was found between the occasion 1 scores of the two groups, t(62) = -.18, p

= .86. A t-test also confirmed there was no significant difference between the

occasion 2 scores of the two groups, t(62) = -.22, p = .83.

A mixed-groups, repeated measures factorial ANOVA was performed to

examine the effects of the use or non-use of a navigation map on problem solving

strategy retention and measured through a problem solving strategy retention test on

occasion 1 and occasion 2. Table 18 shows the means for the conditions of the

design. There was no interaction between treatment and occasion F(1,62) = .00, p

= .96. There was also no main effect for group, F(1,62) = .71, p = .82. There was no

main effect of occasion, F(1,62) = 2.41, p = .13, with no differences in problem

196


solving strategy retention scores in occasion 1 than in occasion 2, for both the control

group and the navigation map group.

Table 18Means for Problem Solving Strategy Retention by Group by OccasionGroup PSS Retention

Occasion 1PSS Retention

Occasion 2Control (n = 31) 6.19 (3.48) 5.71 (2.76) 5.95 (3.12)Treatment (n = 33) 6.06 (2.30) 5.55 (3.15) 5.81 (2.73)

6.13 (2.89) 5.63 (2.96)

Transfer question. As shown in Table 19, the mean scores for the problem

solving strategy transfer occasion 1 and occasion 2 responses for the control group

were 2.90 and 2.26, respectively. The mean scores for the problem solving strategy

transfer occasion 1 and occasion 2 responses for the navigation map group were 3.00

and 2.36, respectively. The mean scores for the problem solving strategy transfer

occasion 1 and occasion 2 responses for the both groups combined were 2.95 and

2.31, respectively. Mean scores for improvement from occasion 1 to occasion 2 for

the control group, the navigation map group, and both groups combined were -.29,

-.48, and -.39, respectively.

197


Table 19Descriptive Statistics of Problem Solving Strategy Transfer Occasion 1 and Occasion 2 Scores for the Control Group, Navigation Map Group, and Both Groups Combined


Occasion 1 2.90 1.70Occasion 2 2.26 1.73Improvement -.65 1.60

Navigation Map (n = 33)Occasion 1 3.00 2.02Occasion 2 2.36 1.93Improvement -.64 1.82

Total (n = 64)Occasion 1 2.95 1.86Occasion 2 2.31 1.83Improvement -.64 1.70


strategy transfer between the control group and the navigation map group, t(62) =



negligible effect. The occasion 1 and occasion 2 problem solving strategy transfer

scores for the control group were significantly correlated, r = .56, p < .01, as were

those of the navigation map group, r = .58, p < .01. A t-test also confirmed that no

significant difference was found between the occasion 1 scores of the two groups,

t(62) = -.21, p = ..84. A t-test also confirmed there was no significant difference

between the occasion 2 scores of the two groups, t(62) = -.23, p = .82.

Another way to look at the problem solving strategy retention data is to

calculate the percentage of the mean scores for problem solving strategy transfer by

the control group, the treatment group, and both groups combined compared to the

number of expert idea units created for the problem solving strategy transfer

198


question. The experts defined 21 idea units related to the problem solving strategy

transfer question (see Table 6). The percentage of a group’s means score is

calculated by dividing the group’s mean score by 21—the number of expert problem

solving strategy retention idea units. For example, the mean score for the control

group for occasion 1 was 2.90 (see Table 19). Dividing that score by the number of

expert idea units (2.90 divided by 21) yielded a mean percentage of 13.81% for the

control group for occasion 1. That is, the control group’s mean score for the total

number of problem solving strategy transfer idea units generated is equal to 13.81%

of the total number of expert problem solving strategy transfer idea units.

As seen in Table 20, mean percentages for the control group were 13.81% for

the occasion 1 and 10.76% for occasion 2. Mean percentages for the navigation map

group were 14.29% for occasion 1 and 11.24% for occasion 2. Mean percentages for

both groups combined were 14.05% for occasion 1 and 11.00% for occasion 2.

Table 20Descriptive Statistics of the Percentage of Problem Solving Strategy Transfer Occasion 1 and Occasion 2 Scores for the Control Group, Navigation Map Group, and Both Groups Combined


Occasion 1 13.81 % 8.10%Occasion 2 10.76% 8.24%Improvement -3.05% 7.62%

Navigation Map (n = 33)Occasion 1 14.29% 9.62%Occasion 2 11.24% 9.19%Improvement -3.05% 8.67%

Total (n = 64)Occasion 1 14.05% 8.86%Occasion 2 11.00% 8.71%Improvement -3.05% 8.10%

199



strategy transfer between the control group and the navigation map group, t(62) =



negligible effect. The occasion 1 and occasion 2 problem solving strategy transfer

scores for the control group were significantly correlated, r = .56, p < .01, as were

those of the navigation map group, r = .58, p < .01. A t-test also confirmed that no

significant difference was found between the occasion 1 scores of the two groups,

t(62) = -.21, p = ..84. A t-test also confirmed there was no significant difference

between the occasion 2 scores of the two groups, t(62) = -.23, p = .82.


examine the effects of the use or non-use of a navigation map on problem solving

strategy understanding as measured through the problem solving strategy transfer

test on occasion 1 and occasion 2. Table 21 shows the means for the conditions of

the design. There was no interaction between treatment and occasion F(1,62) = .21, p

= .65. There was also no main effect for group, F(1,62) = .10, p = .75. There was a

main effect of occasion, F(1,62) = 3.33, p = .07, with no differences in problem

solving strategy transfer scores in occasion 1 than in occasion 2, for both the control

group and the navigation map group.

Table 21Means for Problem Solving Strategy Transfer by Group by OccasionGroup PSS Transfer

Occasion 1PSS TransferOccasion 2


2.95 (1.71) 2.31 (1.83)

200


Trait Self-Regulation Measure

Participants’ trait self-regulation was assess through a self-report instrument

developed by O’Neil and Herl (1998). The trait self-regulation questionnaire

Appendix A) contained 32 questions, eight each to evaluate the four self-regulation

traits represented in the O’Neil Problem Solving model (O’Neil, 1999): planning,

self-monitoring, mental effort, and self-efficacy.

Table 22 shows the mean scores for the four self-regulation factors for the

control group and the navigation map group. Mean scores for the four factors for the

control group were 24.87, 24.03, 23.74, and 25.55 for planning, self-monitoring,

effort, and self-efficacy, respectively. Mean scores for the navigation map group

were, 25.03, 23.91, 24.12, and 24.73 for the same order of factors. As expected, T-

tests confirmed no significant difference by group was found between any of the

self-regulation measures: planning, t(62) = -.19, p = .85; self-monitoring, t(62) =

-.12, p = .91; effort, t(62) = .41, p = .69; self-efficacy, t(62) = -.76, p = .45.

Table 22Descriptive Statistics of Trait Self-Regulation Scores for the Control Group and Navigation Map Group

Scale

Control (n = 31)

Navigation Map(n = 33)

M SD M SDPlanning 24.87 4.06 25.03 2.37Self-Monitoring 24.03 4.70 23.91 3.58Effort 23.74 3.92 24.12 3.53Self-Efficacy 25.55 4.85 24.73 3.74

An analysis of correlations was conducted between the four factors of the

trait self-regulation questionnaire scores and the knowledge map scores, against

201


problem solving strategy retention scores and problem solving strategy transfer

scores. Tables 23, 24, and 25 show the correlations for the control group. Tables 26,

27, and 28 show the correlations for the navigation map group. Tables 29, 30, and 31

show the correlations for both groups combined. As can be seen in Tables 23 and 24,

for the control group, there was no significant relationship between self-regulation

and either knowledge mapping performance or problem solving strategy retention.

However, as can be seen in Table 25, for the control group, there was a negative

correlation between planning ability and the amount of improvement in the problem

solving strategy transfer scores, with greater planning ability leading to poorer

problem solving strategy transfer performance , r = -.43, p < .05.

As can be seen in Tables 26 and 28, for the navigation map group, there was

not significant relationship between self-regulation and either knowledge mapping

performance or problem solving strategy transfer. However, as can be seen in Table

27, for the navigation map group, there was a positive correlation between mental

effort and the amount of improvement in the problem solving strategy retention

scores, with greater mental effort leading to greater problem solving strategy

retention, r = .38, p <.05. As can be seen in Tables 29, 30, and 31, for both groups

combined, there were not significant correlations between self-regulation and

knowledge mapping performance, problem solving strategy retention, or problem

solving strategy transfer.

202


Table 23Correlation Between Self-Regulation Components and Occasion 1, Occasion 2, and Improvement for Knowledge Maps for the Control Group

KMOccasion 1

KMOccasion 2

KMImprovement

Planning .29 .22 -.03Self-Monitoring -.01 -.02 -.02Effort .24 .16 -.06Self-Efficacy .20 .19 .02

Table 24Correlation Between Self-Regulation Components and Occasion 1, Occasion 2, and Improvement for Problem Solving Strategy Retention Responses by the Control Group

PSSRetention

Occasion 1

PSSRetention

Occasion 2

PSSRetention

ImprovementPlanning .15 .04 -.14Self-Monitoring -.04 -.07 -.02Effort .19 .09 -.14Self-Efficacy .29 .27 -.10

Table 25Correlation Between Self-Regulation Components and Occasion 1, Occasion 2, and Improvement for Problem Solving Strategy Transfer Responses by the Control Group

PSSTransfer

Occasion 1

PSSTransfer

Occasion 2

PSSTransfer

ImprovementPlanning .21 -.19 -.43*Self-Monitoring -.04 -.29 -.28Effort .17 -.02 -.20Self-Efficacy .08 .03 -.05

* p < .05

Table 26Correlation Between Self-Regulation Components and Occasion 1, Occasion 2, and Improvement for Knowledge Maps for the Navigation Map Group

KMOccasion 1

KMOccasion 2

KMImprovement

Planning -.20 -.01 .16Self-Monitoring .12 .16 .08Effort -.16 .18 .34Self-Efficacy .09 .22 .17

203


Table 27Correlation Between Self-Regulation Components and Occasion 1, Occasion 2, and Improvement for Problem Solving Strategy Retention Responses by the Navigation Map Group

PSSRetention

Occasion 1

PSSRetention

Occasion 2

PSSRetention

ImprovementPlanning .13 .19 .13Self-Monitoring -.07 .04 .12Effort .03 .29 .38*Self-Efficacy -.06 .04 .12

* p < .05

Table 28Correlation Between Self-Regulation Components and Occasion 1, Occasion 2, and Improvement for Problem Solving Strategy Transfer Responses by the Navigation Map Group

PSSTransfer

Occasion 1

PSSTransfer

Occasion 2

PSSTransfer

ImprovementPlanning .25 -.02 .25Self-Monitoring -.12 -.18 -.06Effort .10 .30 .21Self-Efficacy .27 .34 .07

Table 29Correlation Between Self-Regulation Components and Occasion 1, Occasion 2, and Improvement for Knowledge Maps for Both Groups Combined

KMOccasion 1

KMOccasion 2

KMImprovement

Planning .13 .14 .04Self-Monitoring .04 .05 .02Effort .07 .17 .14Self-Efficacy .16 .20 .08

204


Table 30Correlation Between Self-Regulation Components and Occasion 1, Occasion 2, and Improvement for Problem Solving Strategy Retention Responses for Both Groups Combined

PSSRetention

Occasion 1

PSSRetention

Occasion 2

PSSRetention

ImprovementPlanning .14 .10 -.05Self-Monitoring -.05 -.02 .04Effort .12 .19 .08Self-Efficacy .17 .16 -.01

Table 31Correlation Between Self-Regulation Components and Occasion 1, Occasion 2, and Improvement for Problem Solving Strategy Transfer Responses for Both Groups Combined

PSSTransfer

Occasion 1

PSSTransfer

Occasion 2

PSSTransfer

ImprovementPlanning .02 -.12 -.14Self-Monitoring -.08 -.24 -.17Effort .13 .14 .02Self-Efficacy .17 .18 .01

Safe Cracking Performance

The problem solving performance outcomes for this study are based on the

O’Neil Problem Solving model (O’Neil, 1999), which defines problem solving in

terms of content understanding, problem solving strategies, and self-regulation (see

Figure 1). Problem solving strategy was measured with retention and transfer

questions similar to the methodology employed by Mayer (e.g., Mayer et al., 2002).

An alternative view of problem solving strategy outcomes in this study might be the

number of safes opened by participants. There were a possible 5 safes to open in

game 1 and 5 safes to open in game two. However, two of the safes in game 2 also

appeared in game 1. If a participant opened those safes in game 1, he or she was

205


likely to open them quickly in game 2. Twenty participants opened one of the those

two safes in game 1 with forty-three opening that safe in game 2. Therefore, 23

participants who did not open that safe in game 1 opened it in game 2. For the other

of the two safes, twenty participants opened that safe in game 1 with thirty-eight

opening the safe in game 2. Therefore, 18 participants who did not open that safe in

game 1 opened it in game 2.

While in actuality, there were only 8 different safes in the two games

combined, each game involved opening 5 safes, for a total of 10 safe. Further, with

regards to the two safes that appeared in both games, those who opened them in the

first game also opened them in the second game, getting credit twice for the same

safe. While this skews the results, alternative approaches would also skew results.

For example, if participants were to only get credit once for opening one of the two

common safes, in which game do they get credit; game 1 or game 2? If game 1, then

the most safes they could get credit for in game 2 would be 3 safes. And if instead

credit was given in game 2, the most the participant could receive credit for in game

1 would be 3 safes. In either case, the participant score would incorrectly reflect

performance. The decision was made to give the participant credit for both games,

since it seemed fairer to give participants credit for safes opened, even if given credit

twice, than to not be given credit for safes opened. It should also be noted that, for

game 1, the experimenter opened two safes—the safes in the Reception room. And in

game 2, the experimenter opened four safes—the two safes in the Reception room

and the two safes in the Small Showroom.

206


Table 32 shows the mean scores for the number of safes opened during each

occasion by group, as well as the means scores for the total number of safes opened

by the control group and the navigation map group. For the control group, as shown

in Table 32, the mean scores for the number of safes opened in occasion 1 and

occasion 2 were 2.68 and 2.32, respectively. For the navigation map group, the mean

scores for the number of safes opened in occasion 1 and occasion 2 were 2.70 and

2.21, respectively. Mean scores for total number of safes opened (occasion 1 plus

occasion 2) for the control group and navigation map group were 4.35 and 4.33,

respectively. Note, these scores reflect the safes opened by the participants, and do

not include the two safes opened by the experimenter for game 1 and the four opened

by the experimenter for game 2.

Table 32Descriptive Statistics of the Number of Safes Opened During Occasion 1 and Occasion 2, and the Total Number of Safes Opened by the Control Group, Navigation Map Group, and Both Groups Combined


Game 1 2.68 1.49Game 2 2.32 1.58Total Safes 4.35 2.21

Navigation Map (n = 33)Game 1 2.70 1.31Game 2 2.21 1.29Total Safes 4.33 1.85

Total (n = 64)Game 1 2.69 1.39Game 2 2.27 1.43Total Safes 4.34 2.02

There was no significant difference in the number of safes opened during the

first occasion by the control group and the navigation map group, t(62) = .06, p

207


= .96, Cohen’s d effect size index = .01. The effect size of .01 for the differences in

the number of safes opened by the control and navigation groups during occasion 1

indicated a negligible effect. There was no significant difference in the number of

safes opened during the second occasion by the control group and the navigation

map group, t(62) = -.31, p = .76, Cohen’s d effect size index = .08. The effect size

of .08 for the differences in the number of safes opened by the control and navigation

map groups during occasion 2 indicated a negligible effect. There was no significant

difference in the total number of safes opened during both occasion 1 and occasion 2

by the control group and the navigation map group, t(62) = -.04, p = .97, Cohen’s d

effect size index = .01. The effect size of .01 for the differences in the total number

of safes opened by the control and navigation groups indicated a negligible effect.


examine the effects of the use or non-use of a navigation map on performance as

measured through the number of safes opened on occasion 1 and occasion 2. Table

33 shows the means for the conditions of the design. There was no interaction

between treatment and occasion F(1,62) = .15, p = ..70. There was also no main

effect for group, F(1,62) = .07, p = .89. There was a main effect of occasion, F(1,62)

= 6.28, p = < .05, with more safes opened in occasion 1 than in occasion 2, for both

the control group and the navigation group.

Table 33Means for the Number of Safes Opened by Group by OccasionGroup PSS Transfer

Occasion 1PSS TransferOccasion 2


2.69 (1.40) 2.27 (1.44)

208


Correlations were also generated for the number of safes opened in the first

and second occasions and the total number of safes opened (the number of safes

from the first occasion plus the number of safes from the second occasion) for the

control group (Table 34), the navigation map group (Table 35), and both groups

combined (Table 36). For both groups combined, there was a significant negative

relationship between amount of mental effort and the number of safes opened in the

first game, with more mental effort resulting in less safes opened, r = -.25, p < .05.

For the navigation map group, the same negative relationship between mental effort

and number of safes opened in the first game was found, with more mental effort

results in less safes opened, r = -.38, p < .05. This negative relationship was not

found for the control group, r = -.15, p = .43.

For both groups combined, a positive relationship was found between self-

efficacy the number of safes opened in the second game, with more self-efficacy

resulting in more safes opened, r = .26, p < .05. This relationship was not found for

either the control group (r = .29, p = .11) or the navigation map group (r = .21, p

= .23). As expected, t-tests confirmed no significant difference by group was found

between any of the self-regulation measures: planning, t(62) = -.19, p = .85; self-

monitoring, t(62) = -.12, p = .91; effort, t(62) = .41, p = .69; self-efficacy, t(62) =

-.76, p = .45.

209


Table 34Correlation Between Self-Regulation Components and Number of Safes Opened by the Control Group

Safes Opened in First Game

Safes Opened inSecond Game

Total number ofSafes Opened

Planning -.13 .10 .01Self-Monitoring -.15 .02 -.08Effort -.15 .09 -.08Self-Efficacy .07 .29 .19

* p < .05

Table 35Correlation Between Self-Regulation Components and Number of Safes Opened by the Navigation Map Group




Planning -.06 -.15 -.15Self-Monitoring -.31 -.26 -.30Effort -.38* .02 -.10Self-Efficacy .10 .21 .19

* p < .05

Table 36Correlation Between Self-Regulation Components and Number of Safes Opened for Both Groups Combined




Planning -.10 .01 -.04Self-Monitoring -.22 -.09 -.17Effort -.25* .06 .09Self-Efficacy .08 .26* .19

* p < .05

Continuing Motivation Measure

The term continuing motivation is defined by Malouf (1987-1988) as

returning to a task or a behavior without apparent external pressure to do so when

other appealing behaviors are available. Similarly, Story and Sullivan (1986)

commented that the most common measure of continuing motivation is whether a

210


student returns to the same task at a later time. Because continuing motivation

requires returning to a task, an extra half hour was set aside for participants to

continue playing SafeCracker, if they chose to do so. Further, because continuing

motivation would require continuing to play when other appealing behaviors were

available (Malouf, 1987-1988), indicating a desire to continue playing but not

actually continuing to play was not considered exhibition of continuing motivation.

Participants received a score of one if they continued playing, regardless of the

amount of time they continued to play, and a score of zero if they didn’t continue

playing.

Table 37 shows the mean scores for continuing motivation for both the

control group and the navigation map group. The mean score for continuing

motivation for the control group was .10, while the mean score for the navigation

map group was .15. The amount of continuing motivation was not significantly

different between the two groups, t(62) = .65, p = .52, Cohen’s d effect size index

= .17. The effect size of .17 for the differences in continuing motivation by the

control and navigation groups indicated a negligible effect.

Table 37Descriptive Statistics of the Continuing Motivation Scores of the Control Group, Navigation Map Group, and Both Groups Combined


Continuing Motivation .10 .30Navigation Map (n = 33)

Continuing Motivation .15 .36Total (n = 64)

Continuing Motivation .13 .33

211


Tests of the Research Hypotheses




Tables 12 and 13 showed that the navigation map group did not have

significantly greater content understanding than the control group as measured by

knowledge map construction. While the amount of improvement for the navigation

map group (M = 1.34) was greater than the amount of improvement for the control

group (M = .98), the difference was not significant. Hypothesis 1 was not supported.




Tables 16 and 17 showed that the navigation map group did not retain more

problem solving strategies than the control group. For both groups, retention

decreased from occasion 1 to occasion 2. While the decrease in retention was more

pronounced for the control group (M = -.52) than for the navigation map group (M =

-.42), the difference between the two groups was not significant. Hypothesis 2 was

not supported.

Hypothesis 3: Participants who use a navigation map (the treatment group) will

exhibit greater problem solving strategy transfer than participants who do not use a

navigation map (the control group).

Tables 18 and 19 showed that the navigation map group did not exhibit more

problem solving strategy transfer than the control group. For both groups, transfer

212


decreased from occasion 1 to occasion 2. While the decrease was more pronounced

for the navigation group (M = -.48) than for the control group (M = -.29), the

difference between the groups was not significant. Hypothesis 3 was not supported.




better performance.

As indicated by Table 22, there were no significant differences in the self-

regulation scores between the control group and the navigation map group. The mean

scores for planning, self-monitoring, effort, and self-efficacy were 24.87, 24.03,

23.74, and 25.55, respectively, for the control group and 25.03, 23.91, 24.12, and

24.73, respectively, for the navigation map group.

The latter part of the hypotheses, that higher levels of self-regulation would

be associated with better performance, was only partially supported. With regards to

knowledge mapping (Tables 23, 26, and 29) and problem solving strategy retention

(Tables 24, 27, and 30) and problem solving strategy transfer (Tables 25, 28, and 31)

two correlations existed between performance and self regulation. As shown in Table

25, a negative correlation between planning ability and the amount of improvement

in the problem solving strategy transfer scores, r = -.43, p < .05. As shown in Table

27, a positive correlation between amount of mental effort and the amount of

improvement in the problem solving strategy retention scores, r = .38, p <.05.

Another indicator of performance was the number of safes opened (Tables 32

and 33). There were no differences between the mean scores of the two groups in

213


number of safes opened in occasion 1, occasion 2, or the total number of safes

opened. Several correlations were found between number of safes opened and self-

regulation measures. For both groups combined (Table 36), there was a negative

correlation between mental effort and the number of safes opened in game 1 (r =

-.25, p < .05). As shown in Table 35, the same negative correlation was found in the

navigation group (r = -.38, p < .05). Table 36 also shows a positive correlation

between self-efficacy and the number of safes opened in game 2 for both groups

combined (r = .26, p < .05). There were no correlations between the control group

and self-regulation scores (Table 34). Hypothesis 4 was only partially supported.

Hypothesis 5: Participants who use a navigation map (the treatment group) will

exhibit a greater amount of continuing motivation, as indicated by continued optional

game play, than participants who do not use a navigation map (the control group).

Table 37 shows the mean scores for continuing motivation for the control

group (M = .10) and the navigation map group (M = .15). While the continuing

motivation score for the navigation map group was higher than the control group’s

score, the difference between the two groups was not significant, t(62) = .65, p = .52.

Hypothesis 5 was not supported.

214


CHAPTER 5

SUMMARY OF THE RESULTS AND DISCUSSION

The purpose of this study was to examine the effect of a navigation map on a

complex problem solving task in a 3-D, occluded, computer-based video game. With

one group playing the video game while using a navigation map (the treatment

group) and the other group playing the game without aid of a navigation map (the

control group), this study examined differences in problem solving outcomes as

informed by the O’Neil (1999) Problem Solving model. The O'Neil model delineated

problem solving into content understanding, problem solving strategies, and self-

regulation. Five hypotheses were generated for this study. The first four addressed

the three components of the O’Neil Problem Solving model, asserting that those who

used a navigation map (the treatment group) would exhibit greater content

understanding (hypothesis 1), greater retention of problem solving strategies

(hypothesis 2), and greater transfer of problem solving strategies (hypothesis 3) than

those who did not use a navigation map (the control group). The fourth hypothesis

asserted that those with higher amounts of trait self-regulation would perform better

than those with lower amounts of trait self-regulation. The fifth hypothesis of the

study asserted that those who used the navigation map (the treatment group) would

exhibit greater continuing motivation than those who did not use the navigation map

(the control group), as exhibited by continued optional play of the game.

Summary of the Results

Results of the data analysis indicated that the use of navigation maps did not

affect problem solving as measured by performance based to the O’Neil (1999)

215


Problem Solving model. Those using the navigation map (the treatment group) did

not score higher than those who did not use the navigation map (the control group) in

content understanding, problem solving strategy retention, and problem solving

strategy transfer. In addition, with some minor exceptions, higher levels of self-

regulation were unrelated to higher levels of performance regardless of whether or

not a map was used. Lastly, those who used the navigation map (the treatment group)

did not exhibit higher continuing motivation than those who did not use the map (the

control group).

While the results of the data analysis in this study did not provide support any

of the study’s five hypotheses, examination of the results may provide insights not

only into why these results occurred in this study but to characteristics of game-

based problem solving environments and navigation maps that may inform the field

and affect not only future studies, but game design and instructional design as well.

While the purpose of this study was to examine the effect of navigation maps on

performance outcomes, other factors may have contributed to the lack of sufficient

influence by the navigation maps.

To explain the lack of statistical difference between the treatment group

(navigation map) and control group (no map), two effects should be examined: one

that would reduce or suppress the effects of the treatment (the navigation map) and

one that would inflate the outcomes measures for the control group. Since the

hypotheses of this study are based on the cognitive load and graphical scaffolding

research and theories of Richard Mayer and John Sweller, plausible explanations

should fit within those frameworks. Suppression of outcomes by the treatment group

216


might well be explained by extraneous load theory and by the contiguity effect.

Inflation of the control group might well be explained by priming.

The contiguity effect proposes that separating items of importance either

spatially or temporally adds cognitive load. Since the navigation map was separated

from the game, spatial contiguity may have contributed to cognitive overload.

Extraneous load theories purport that attending to unnecessary items adds cognitive

load unrelated to the task, reducing the amount of cognitive capacity available for

processing necessary information or even contributing to cognitive overload. The

complex 3-D environment of the game SafeCracker was filled with visual and other

details that would be expected to add extraneous cognitive load. If the task of

navigating to rooms was not as cognitively challenging as expected, the addition of

the navigation map may have been detrimental, rather than beneficial.

Priming asserts that providing cues can help focus attention on important

tasks or details, which ultimately helps with metacognitive process involved in

learning and problem solving. Both groups were primed a number of times with

search and problem solving strategies, which might have aided both groups in

understanding procedures necessary for doing well in the SafeCracker. Those

strategies may have influenced both groups enough to offset any differences that

might have been fostered by navigation map usage, ultimately resulting in similar

outcomes for the two groups. This chapter is divided into three sections. First will be

a discussion of possible explanations based on the contiguity effect and extraneous

load. Second will be a discussion of possible explanations based on strategy priming.

Last will be a summary of the discussions.

217


Discussion

Possible Effects from the Contiguity Effect and Extraneous Load

A major instructional issue in learning by doing within simulated

environments concerns the proper type of guidance (i.e. scaffolding; Mayer et al.

2002). Mayer and colleagues (2002) commented that scaffolding is an effective

instructional strategy and that discovery-based learning environments can become

effective learning environments when the nature of the scaffolding is aligned with

the nature of the task, such as pictorial scaffolding for pictorially-based tasks and

textual scaffolding for textually-based tasks.

However, while graphical scaffolding appears to be beneficial, there are

potential problems associated with use of this type of scaffolding. One such problem

is termed the contiguity effect, which refers to the cognitive load imposed when

multiple sources of information are separated (Mayer et al., 1999; Mayer & Moreno,

2003; Mayer et al., 1999; Mayer & Sims, 1994; Moreno & Mayer, 1999). There are

two forms of the contiguity effect: spatial contiguity and temporal contiguity.

Temporal contiguity occurs when one piece of information is presented prior to other

pieces of information. Spatial contiguity occurs when information is physically

separated (Mayer & Moreno, 2003). The contiguity effect results in split attention

(Moreno & Mayer, 1999).

According to the split attention effect, when information is separated by

space or time, the process of integrating the information may place an unnecessary

strain on limited working memory resources (Atkinson et al., 2000; Mayer, 2001;

Tarmizi & Sweller, 1998). When dealing with two or more related sources of

218


information (e.g., text and diagrams), it’s often necessary to integrate mentally

corresponding representations (e.g., verbal and pictorial) to construct a relevant

schema to achieve understanding. When the sources of information are separated in

space or time, this process of integration may place an unnecessary strain on limited

working memory resources, resulting in impairment in learning (Atkinson et al.,

2000; Mayer & Moreno, 1998; Tarmizi & Sweller, 1988). The current study likely

imposed spatial contiguity, since the navigation map was presented on a piece of

paper which, depending on where the participant placed the map, was separated from

the computer screen. This study did not examine the impact of this additional

cognitive load; how it might have influenced problem solving outcomes, possibly

adding sufficient cognitive load to offset the cognitive load benefits expected from

the graphical scaffolding (the navigation map).

Extraneous load refers to the cognitive load imposed by unnecessary

(extraneous) materials (Harp & Mayer, 1998; Mayer, Heiser, & Lonn, 2001; Moreno

& Mayer, 2000; Renkl & Atkinson, 2003; Schraw, 1998). Seductive details, a

particular type of extraneous details, are highly interesting but unimportant elements

or instructional segments that are often used to provide memorable or engaging

experiences (Mayer et al., 2001; Schraw, 1998). The seductive detail effect is the

reduction of retention caused by the inclusion of extraneous details (Harp & Mayer,

1998) and affects both retention and transfer (Moreno & Mayer, 2000).

Extraneous cognitive load (Renkl & Atkinson, 2003) is the most controllable

load, since it is caused by materials that are unnecessary to instruction. However,

those same materials may be important for motivation. Some research has proposed

219


that learning might benefit from the inclusion of extraneous information. Arousal

theory suggests that adding entertaining auditory adjuncts will make a learning task

more interesting, because it creates a greater level of attention so that more material

is processed by the learner (Moreno & Mayer, 2000). A possible solution to the

conflict of the seductive detail effect, which proposes that extraneous details are

detrimental, and arousal theory, which proposes that seductive details in the form of

interesting auditory adjuncts may be beneficial, is to include the seductive details,

but guide the learner away from them and to the relevant information (Harp &

Mayer, 1998). SafeCracker is a visually rich and immersive 3-D game environment

that, as with virtually any modern 3-D game, is fraught with extraneous and

seductive details—so much so, that guiding the player away from these details may

be impossible.

The point-of-view of the participants in SafeCracker is that they are standing

in the rooms of a mansion. Participants can “look” around, can “walk” to various

locations in a room, “open” doors, “enter” other rooms, and “pick up” and “look at”

books, pieces of paper, and a variety of other items. Participants attempt to “open”

safes, by interacting with the safes’ locking and opening mechanisms (solving

puzzles), by “looking at” items contained in the participants’ inventories, and by

attempting to “use” objects, such as keys or coins to open the safes. While all of

these details make for a rich, visual interactive experience and for engaging

participants in the environment, they are extraneous to the two major goals of the

problem solving task—“finding” clues, rooms, and safes, and “opening” safes.

Because of the impact of the scope of extraneous details in SafeCracker and because

220


of an inability to draw participants away from those extraneous details to focus on

relevant details, as Moreno and Mayer (2002) suggested, extraneous detail effects

might very well explain the lack of significant differences between the performance

of the treatment and control groups; The extraneous and seductive details may have

placed enough extraneous cognitive load to offset the cognitive load benefits of the

navigation map.

In addition to the extraneous nature of the game environment, the navigation

map itself may have been an extraneous detail. The studies conducted by Chou and

colleagues (Chou & Lin, 1998; Chou et al., 2000) had provided the impetus for this

study. In line with the scaffolding and cognitive load research of Mayer and Sweller

(e.g., Atkinson et al., 2000; Mayer, 2001; Mayer & Moreno, 1998; Tarmizi &

Sweller, 1998), Chou and Lin (1998) examined the use of three map types: global

map, local map, and no map. The global map displayed the entire environment (a

hypertext, node-based environment), while the local map displayed only a portion of

the environment. Chou and Lin (1998) found that knowledge map creation by those

who used a global navigation map in a search related problem solving task was

significantly better than by those who used either a local navigation map or no map.

The navigation map in this study displayed the whole environment (the

entire bottom floor of the mansion) and, thus, would be defined as a global map.

Therefore, the results of this study should have matched the results of the Chou and

Lin (1998) study—but they didn’t. However, results of this study did match the

results of the Chou et al. (2000) study, which found no differences in knowledge

map creation based on map type. Since results from that second Chou and colleagues

221


study differed from the results of other graphical scaffolding studies, including the

earlier Chou and colleague study, had been assumed, prior to this study, that the

results had either been an anomaly or the second Chou and colleagues study had

been flawed. However, it was also possible the second study wasn’t flawed and it

was the nature of the Chou environment that resulted in the mixed results based on

map type (global, local, and no map).

It is possible that the Chou environment (Chou & Lin, 1998; Chou et al.,

2000) was not complex enough need, or benefit from, a navigation map. If that were

the case, then the navigation map would have been an extraneous detail and any

cognitive load benefits of map usage might have been offset by the additional

cognitive load introduced by the presence and use of the map. This could explain the

mixed results of the two Chou and colleagues studies. It could also explain the

results of this study. For this study, it had been believed that the search portion of the

problem solving task of finding and opening safes was sufficiently difficult enough

to require, and to benefit from, use of a navigation map. However, it is quite possible

that was not the case. If the navigation map were unnecessary, then adding the map

would have simply added extraneous cognitive load for the treatment group,

resulting in poorer performance than expected; the benefits from using the map

would have been negated by the additional extraneous cognitive load.

In summary, inclusion of the navigation map may not have provided the

cognitive benefits expected from the inclusion of graphical scaffolding. The

navigation map was separated from the main gaming environment (the computer

screen), which may have resulted in the contiguity effect and the split attention

222


effect, which would cause additional cognitive load. In addition, the general nature

of the SafeCracker environment might have been sufficiently filled with extraneous

details as to offset any benefits from use of a navigation map. Lastly, the search

portion of the problem solving task of searching for and opening safes may not have

been sufficiently difficult to benefit from use of a navigation map. If so, rather than

providing cognitive benefit, the navigation map may have acted as an extraneous

detail, resulting in reduced performance.

Possible Effects from Strategy Training

Priming is a cognitive phenomenon where a stimulus (e.g., word or sound)

readies the mind to allow or engage particular relevant schema. This timely exposure

to stimuli results in enhanced access to stored stimuli or information (retrieved

October 7, 2005 from http://filebox.vt.edu/8080/users/dereese2/module8/

module08bkup/IDProjectWebpage/lesson4.htm). According to Dennis and Schmidt

(2003), repetition priming is closely allied to skill acquisition. Moreno and Mayer

(2005) commented that lack of priming (in the form of guidance) will result in

reduction of the metacognitive process of selecting—one of the key components in

meaningful learning.

In this study, all subjects were primed a number of times and in several key

knowledge areas. One occurrence of priming occurred during knowledge map

training. A series of primings occurred during SafeCracker training. Another

sequence of priming occurred during navigation map training for the treatment group

and during navigation training for the control group. Additional priming occurred at

the start of each of the two SafeCracker games. It is believed that these primes might

223


have improved the skills and game play knowledge of both groups enough to offset

any gains that were expected due to treatment. These priming events could have

inflated the control group’s skills and understanding of the game sufficiently enough

to have offset any differences that might have been seen due to treatment (use of a

navigation map). While priming occurred for the treatment group as well, the

priming might have been more important than use of the navigation map, negating

differences due to navigation map use and resulting in equivalent performance by the

two groups (treatment and control).

The main reason priming was included in this study was to emulate priming

provided in earlier game-based studies that utilized a game entitled Space Fortress

(see Day et al., 2001, for a description of Space Fortress). Numerous studies were

conducted using Space Fortress and each study began by teaching participants how

to play the game and included strategy instructions based on expert player

knowledge and experience (e.g., Day et al., 2001; Gopher et al., 1994; Shebilske et

al., 1992). The purpose of the training was to ensure that every participant began the

game with equivalent game knowledge and playing skills. That way, it could be

assumed that any differences in performances would be attributed to treatment, not

prior abilities or knowledge, or other game-related individual differences. The same

was expected to be true for this study. A secondary reason for adding priming in this

study was a reaction to observations during the pilot study, where neither of the

participants searched for clues. It was decided that priming related to searching for

clues was necessary for the main study. The following describe priming during the

various phases of the study.

224


Strategy priming during knowledge map training. During knowledge map

training, all participants were told that, since every concept was applicable to

SafeCracker, they should add all the concepts to the screen and then begin making

links. These knowledge mapping instructions provided two key elements of priming.

First, participants were told that “all” concepts were to be used and should be added

to their knowledge map. This meant that, if they followed that strategy, early on

during map development they would see all concepts and know that links were

needed for all concepts. Therefore, it would prompt the participants to think about

each combination of concepts, possibly more than they would have. Second, as

participants exhausted the links they were aware of, they were primed by any

concept not involved in a link that a link was missing. This might have fostered

deeper levels of thinking, simply by knowing that a link was missed, was not

obvious to the participant, and needed to be discovered.

Strategy priming during SafeCracker training. From observations during the

pilot study, a number of verbal prompts were added to the SafeCracker instructions

in the main study to assist participants in remembering to search for clues. To

support research that has found that repetition promotes retention, participants were

reminded several times to remember to search for clues. In addition to those

reminders, there were reminders on the importance of various types of clues, and

reminders for participants to write things down, particularly any diagrams they

found. For example, while looking at a piece of paper sitting on a counter in the

game environment, participants were given the instruction, “Notice the diagrams.

These might be important for opening a safe. You might want to write them down

225


later, when you start playing the game.” Not only were participants prompted to go

back to the paper when playing the game, they were primed to the concept that

diagrams could be important (even diagrams not on that particular paper) and that

information should be written down, rather than kept in working or long-term

memory.

Priming during navigation map and basic navigation training. The treatment

group received priming during navigation map training. The control group received

priming during navigation training. The treatment group’s priming included multiple

repetitions of the primes for remembering to search for clues and remembering to

write things down. The control group received similar priming during their

navigation training. While not repeated as often as the primes for the treatment

group, the control group’s priming did repeat earlier priming both groups had

received on searching for clues and writing things down, which should have made

the scope of control group priming and the degree of priming repetition similar to

that of the treatment group.

Priming at the start of each game. At the start of the first game, all

participants were reminded once again to look at objects in the various rooms,

including the room they were currently in, the Reception Room. Prior to beginning

the second game, all participants were reminded once more to search for clues and to

write down any information they deemed important. These repetitions aided in

ensuring that participants remembered and, hopefully, acted on these strategies.

Prior to the second game, all participants were also told that the safes from

two of the rooms from the first game had been opened for them and the contents of

226


the safes in those rooms had been added to their inventories. Participants were then

told they might want to revisit those two rooms from the previous game, to search

for clues they might have missed in the first game. For the control group, this

priming might have caused participants to visit the two rooms from the prior game

which, without the priming, they might not have thought to visit. For the treatment

group, if the navigation map had been effective in reducing cognitive load,

participants might have thought to revisit those rooms, even without the priming. By

contrast, the greater cognitive load the control group was experiencing from lack of a

navigation map may have prevented those participants from making the

determination to revisit those two rooms and search for clues. Therefore, by

providing the strategy, the control group might have been given a strategy they

would not otherwise have had the cognitive capacity to devise.

Overall, a large number of strategy primes were given to both groups

(treatment and control). Priming occurred during knowledge map training and during

SafeCracker training. Priming occurred during navigation map training for the

treatment group and during navigation training for the control group. These

combined primings could have altered the behavior of both groups enough to negate

differences by treatment (navigation map usage) and effectively inflating the

performance outcomes of the control group.

Summary of the Discussion

The lack of significance for any of the hypotheses in this study might be

explained by either something negatively influencing (deflating) performance by the

treatment group or by something positively influencing (inflating) performance by

227


the control group. Two explanations based on the scaffolding and cognitive load

research of Mayer and Sweller have been presented; one to account for deflated

performance by the treatment group and one to account for inflated performance by

the control group. Combined, these two effects provide plausible explanations for the

unexpected results of this study.

For the treatment group, inclusion of the navigation map may not have

provided the cognitive benefits expected from the inclusion of graphical scaffolding.

The navigation map was separated from the main gaming environment (the computer

screen), which may have resulted in the contiguity effect and the split attention effect

for the treatment group. The search portion of the problem solving task of searching

for and opening safes may not have been sufficiently difficult to benefit from use of

a navigation map. If so, rather than providing cognitive benefit, the navigation map

may have acted as an extraneous detail, resulting in reduced performance by the

treatment group. And the general nature of the SafeCracker environment might have

been sufficiently filled with extraneous details to offset any benefits from use of a

navigation map by the treatment group, even if the map were necessary. For the

control group, the large number of strategy primes given to both groups (treatment

and control) could have positively altered behaviors enough to negate differences by

treatment (navigation map usage).

Four of the five study hypotheses were not supported and one was only

partially supported (hypothesis 4). The contiguity and split attention effects, as well

as the effects of extraneous details, appear to be plausible explanations for the results

of all five hypotheses, with regards to deflated treatment group performance.

228


Strategy priming appears to be a plausible explanation for inflating the performance

of the control group and ultimately affecting the results for the five hypotheses.

These combined effects provide reasonable explanation for the results of this study.

229


CHAPTER 6

SUMMARY, CONCLUSIONS, AND IMPLICATIONS

Summary

The purpose of this study was to examine the effect of a navigation map on a

complex problem solving task in a 3-D, occluded, computer-based video game. With

one group playing the video game while using the navigation map (the treatment

group) and the other group playing the game without aid of a navigation map (the

control group), this study examined differences in problem solving outcomes as

informed by the O’Neil (1999) Problem Solving model. The O’Neil model

delineated problem solving into content understanding, problem solving strategies,

and self-regulation.

Five hypotheses were generated for this study. The first four addressed the

three components of the O’Neil (1999) Problem Solving model, asserting that those

who used a navigation map (the treatment group) would exhibit greater content

understanding (hypothesis 1), greater retention of problem solving strategies

(hypothesis 2), and greater transfer of problem solving strategies (hypothesis 3) than

those who did not use a navigation map (the control group). The fourth hypothesis

asserted that those with higher amounts of trait self-regulation would perform better

than those with lower amounts of trait self-regulation. The fifth hypothesis of the

study asserted that those who used the navigation map (the treatment group) would

exhibit greater continuing motivation than those who did not use the navigation map

(the control group), as exhibited by continued optional play of the game.

230


Despite early expectations (Donchin, 1989; Malone, 1981; Malone & Lepper,

1987; Ramsberger, Hopwood, Hargan, & Underfull, 1983; Thomas & Macredie,

1994), research into the effectiveness of games and simulations as educational media

has been met with mixed reviews (de Jong & van Joolingen, 1998; Garris, Ahlers, &

Driskell, 2002; O’Neil, Baker, & Fisher, 2002). It has been suggested that the lack of

consensus can be attributed to weaknesses in instructional strategies embedded in

the media and to other issues related to cognitive load (Chalmers, 2003; Cutmore,

Hine, Maberly, Langford, & Hawgood, 2000; Lee, 1999; Thiagarajan, 1998; Wolfe,

1997). Cognitive load refers to the amount of mental activity imposed on working

memory at an instance in time (Chalmers, 2003; Sweller & Chandler, 1994, Yeung,

1999). Researchers have proposed that working memory limitations can have an

adverse effect on learning (Sweller & Chandler, 1994; Yeung, 1999). Further,

cognitive load theory suggests that learning involves the development of schemas

(Atkinson, Derry, Renkl, & Wortham, 2000), a process constrained by limited

working memory and separate channels for auditory and visual/spatial stimuli

(Brunken, Plass, & Leutner, 2003).

One way to reduce cognitive load is to use scaffolding, which provides

support during schema development by reducing the load in working memory

(Clark, 2001). For example, graphical scaffolding has been shown to provide

effective support for graphically-based learning environments, including video

games (Benbasat & Todd, 1993; Farrell & Moore, 2000; Mayer, Mautone, &

Prothero, 2002). Navigation maps, a particular form of graphical scaffolding, have

been shown to be an effective scaffold for navigation of a three-dimensional (3-D)

231


virtual environment (Cutmore et al., 2000). Navigation maps have also been shown

to be an effective support for navigating in a problem solving task in a two-

dimensional (2-D) hypermedia environment (Baylor, 2001; Chou, Lin, & Sun, 2000).

What has not been examined, and is the purpose of this study, is the effect of

navigation maps, utilized for navigation in a 3-D, occluded, computer-based video

game, on outcomes of a complex problem solving task.

This study utilized an experimental posttest only, 2x2 repeated measures

design with two levels of treatment (maps vs. no maps) and 2 levels of occasion

(occasion 1 vs occasion 2). Participants were randomly assigned to either the

treatment or the control group. The procedure involved administration of pretest

questionnaires, the treatment, the occasion instruments, the treatment, the occasion

instruments, and debriefing. After debriefing, participants were offered up to 30

minutes of additional playing time (to examine continuing motivation). The data for

64 of the participants were included in the data analysis.

A number of instruments were included in the study: a demographic, game

play, and game preference questionnaire, two task completion forms that acted as

advance organizers by listing the names of the rooms to be found and brief

descriptions of the safes in each room, a self-regulation questionnaire the examined

the four self-regulation components of the O’Neil (1999) Problem Solving model

(planning, self-monitoring, mental effort, and self-efficacy), the computer-based

video game SafeCracker®, two navigation map of the game’s environment (each

highlighting the rooms involved in the two games that would be played), a problem

solving strategy retention and transfer questionnaire to be completed after each of the

232


two SafeCracker games; and knowledge mapping software to be completed after

each of the two SafeCracker games.

Results of the study did not support the five hypotheses. With regard to

hypothesis 1, results of the data analysis found that the navigation map group did not

have significantly greater content understanding than the control group as measured

by knowledge map construction. Hypothesis 1 was not supported. With regards to

hypothesis 2, results of the data analysis found that the navigation map group did not

retain significantly more problem solving strategies than the control group.

Hypothesis 2 was not supported. With regards to hypothesis 3, results of the data

analysis found that the navigation map group did not exhibit significantly more

problem solving strategy transfer than the control group. Hypothesis 3 was not

supported.

That higher levels of self-regulation would be associated with better

performance (hypothesis 4) was only partially supported. With regards to knowledge

mapping, problem solving strategy retention, and problem solving strategy transfer,

two correlations existed between performance and self regulation. A negative

correlation between planning ability and the amount of improvement in the problem

solving strategy transfer scores, r = -.43, p < .05. A positive correlation between

amount of effort and the amount of improvement in the problem solving strategy

retention scores, r = .38, p <.05. With regards to hypothesis 5, while the continuing

motivation score for the navigation map group (M=15) was higher than the control

group’s score (M=10), the difference between the two groups was not significant.

Hypothesis 5 was not supported.

233


In summary, results of the data analysis indicated that the use of navigation

maps did not affect problem solving as measured by the performance based to the

O’Neil (1999) Problem Solving model. Those using the navigation map (the

treatment group) did not score higher than those who did not use the navigation map

(the control group) in content understanding, problem solving strategy retention, and

problem solving strategy transfer. In addition, higher levels of self-regulation were

unrelated to higher levels of performance regardless of whether or not a map was

used. Lastly, those who used the navigation map (the treatment group) did not

exhibit higher continuing motivation than those who did not use the map (the control

group).

These results were surprising, as the hypotheses for this study were based on

the work of Richard Mayer (e.g., Mayer et al., 2002) and John Sweller (e.g.,

Tuovinen & Sweller, 1999), which would have predicted support for all hypotheses.

Based on cognitive load theory, an important cognitive goal in design is to control

the amount of load placed on working memory, particularly by items not necessary

for learning. Navigation maps, a graphical form of scaffolding, would serve such a

purpose, by distributing the need to retain location and paths from working memory

to an external graphical support. It appears from this study, though, that such support

may not have been necessary in this game or that the maps did not offer appropriate

or sufficient scaffolding.

To explanations were examined; one that looked at possible causes of

deflated performance by the treatment group (navigation map) and one that looked at

a possible cause of inflated performance by the control group (no map). Specifically,

234


the following were examined: the split attention effect (Mayer, 2001; Tarmizi &

Sweller, 1988) and its related contiguity effect (Mayer et al., 1999; Mayer &

Moreno, 1998; Mayer & Sims, 1994; Moreno & Mayer, 1999; Yeung et al., 1997);

the negative cognitive effects of extraneous and seductive details (Mayer, 1998,

Mayer et al., 2001); and theories related to priming (Dennis & Schmidt, 2003).

Conclusions

Several potential causes were examined to explain the unexpected results of

this study. Overall, it is likely that the results were related to cognitive load. The

contiguity and split attention effects, as well as extraneous cognitive load, seem to

provide plausible explanations for why the treatment group may not have benefited

from the navigation map. Strategy priming seems to provide a plausible explanation

for why the control group might have performed at levels equivalent to the treatment

group.

The separation of the map from the playing area would account for increased

cognitive load for the treatment group, since this separation would have introduced

contiguity and split attention effects. However, it is unclear whether these effects

would have been sufficient to negate greater performance by the treatment group.

Extraneous detail effects, including the effects of seductive details, would also

account for increased cognitive load, but it is unclear whether the effect would have

negated the effects of navigation map use by the treatment group enough to cause

both groups to perform equally. Priming is also a viable explanation of study results,

but it is unknown whether priming could have increased the performance of both

groups enough to negate the effects of navigation map use by the treatment group.

235


Implications

Off the shelf games might provide a platform for some research, but the

constraints imposed by off the self games might preclude many games from being

useful enough as research platforms, due to the lack of control of a number of

variables. One solution would be to use games that include editing abilities

sophisticated enough to modify the game to meet study requirements, including the

ability to add or remove elements, modify existing elements, and collect a variety of

data. A second solution would be to develop a game specifically for a study. While

this method would be the most costly and time consuming, it would offer the

advantage of creating a research environment containing every component needed

for treatment and control groups, enable modification of every element in the game,

from interface, to environment, to controls, to goals, and allow for tracking and

outputting of all desirable data.

The results of this study have shown that use of a navigation map does not

guarantee improvements over not using a navigation map. While scaffolding has

been shown to be a useful and beneficial instructional strategy, graphical scaffolding

has been shown to be an effective aid in graphically-based environments, and

navigation maps have been shown to provide benefits in 3-D space, it is apparent

from the results of this study that factors may exist that preclude benefits in all

situations. Several factors have been presented as possible explanations for the lack

of benefit from navigation map usage. To determine which or which combination of

these factors are the cause or causes of the findings of this study, a series of

experiments should be conducted. A customizable gaming environment must be

236


used, to control for each variable, to introduce each variable one at a time or in

combinations, and to track participant activities. It is through this careful evaluation

of variables that the benefits or limitations of navigation map usage can be

discovered.

More and more, games are being seen as a viable delivery platform for

educational content. But as has been found through a number of studies, it is not the

game but the instructional methods built into the game that result in learning. A

game may provide motivation, but it does not, of itself, provide learning. Only the

methods, content, and strategies embedded in the game can provide that learning. As

a potentially useful instructional strategy, it is important to discover the

circumstances under which navigation maps are beneficial to learning. Only then can

we begin to prescribe their use.

Since treatment had been expected to result in better performance in this

study, but did not, those factors that might have influenced navigation map

performance should be examined first. Three effects need to be tested, all relating to

unnecessary load caused by the use of the navigation map. The contiguity and related

split attention effects can be examined by having the map appear on screen, either on

the game’s interface or near the player’s focal point on the screen. The extraneous

load caused by the complex 3-D game environment can be examined by having

players use environments of varying complexity, from simply colored objects, bare

walls, and basic, boxy furniture to feature rich environments such as the environment

of SafeCracker. The third study would involve examining the use of the navigation

map in environments of varying scope. Because the search portion of the problem

237


solving task may not have been complex enough to benefit from a navigation map,

environments from as small as the single floor of a mansion as used in this study to

environments as large as several building, each with multiple floors, or even

buildings separated by complex occluded paths, such as swamps, lakes, or roads,

should be examined. In addition, the number of rooms involved in the problem

solving task could be varied from three rooms as in this study to ten or more rooms.

To examine the possible effect of priming on performance, priming could be

examined by varying the degree of strategies offered to players, from no strategies to

numerous strategies. It is also suggested that the amount of repetition be varied, to

determine the impact repetition has on priming for problem solving strategies in a

video game. Priming could be examined as a 2-by-3 study, with two levels of

strategy inclusion (none versus some amount) and two levels of repetition (none

versus a small amount versus a large amount).

More and more, educational institutions seem to be embracing the use of

video game and simulation environments as a way of modernizing teaching. The

primary impetus for this change in learning strategy is the belief that the motivational

aspects of games and simulations will lead to improvements in learning. Yet, as

research as shown, it is the quality and appropriateness of the instructional strategies

embedded in learning environment that determine whether or not learning will occur.

Little is known about the use of immersive 3-D games for learning, and this study

highlights the fact that what works in one learning environment may not work in

another. To ensure that 3-D games provide the necessary features to foster learning,

studies examining instructional strategies that have been previously shown to be

238


effective in other learning environments must be carefully examined for

effectiveness in this new learning environment. One such strategy is the use of

navigation maps.

239


REFERENCES

Adams, P. C. (1998, March/April). Teaching and learning with SimCity 2000 [Electronic Version]. Journal of Geography, 97(2), 47-55.

Alessi, S. M. (2000). Simulation design for training and assessment. In H. F. O’Neil, Jr. & D. H. Andrews (Eds.), Aircrew training and assessment (pp. 197-222). Mahwah, NJ: Lawrence Erlbaum Associates.

Alexander, P. A. (1992). Domain knowledge: Evolving themes and emerging concerns. Educational Psychologist, 27(1), 33-51.

Allen, R. B. (1997). Mental models and user models. In M. Helander, T. K. Landauer & P. Prabhu (eds.), Handbook of Human Computer Interaction: Second, Completely Revised Edition (pp. 49-63). Amsterdam: Elsevier

The American Heritage Dictionary of the English Language (fourth edition). (2002). Boston, MA: Houghton Mifflin Co.

Anderson, L. W., Krathwohl, D. R., Airasian, P. W., Cruikshank, K. A., Mayer, R. E., Pintrich, P. R., et al. (Eds.) (2001). A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom’s Taxonomy of Educational Objectives (complete edition). New York, NY: Longman.

Anderson, R. C., & Pickett, J. W. (1978). Recall of previously recallable information following a shift in perspective. Journal of Verbal Learning and Verbal Behavior, 17, 1-12.

Armer, A. (1988). Writing the Screenplay. Belmont, CA: Wadsworth Publishing.

Arthur, W., Jr., Strong, M. H., Jordan, J. A., Williamson, J. E., Shebilske, W. L., & Regian, J. W. (1995). Visual attention: Individual differences in training and predicting complex task performance. Acta Psychologica, 88, 3-23.

Asakawa, T., Gilbert, N. (2003). Synthesizing experiences: Lessons to be learned from Internet-mediated simulation games. Simulation & Gaming, 34(1), 10-22.

Atkinson, R. K., Derry, S. J., Renkl, A., & Wortham, D. (2000). Learning from examples: Instructional principles from the worked examples research. Review of Educational Research, 70(2), 181-214.

Atkinson, R. K., Renkl, A., Merrill, M. M. (2003). Transitioning from studying examples to solving problems: Effects of self-explanation prompts and fading worked-out steps. Journal of Educational Psychology, 95(4), 774-783.

240


Ausubel, D. P. (1963). The psychology of meaningful verbal learning. New York: Grune and Stratton.

Ausubel, D. P. (1968). Educational psychology: A cognitive view. New York: Holt, Reinhart, and Winston.

Baddeley, A. D. (1986). Working memory. Oxford, England: Oxford University Press.

Baddeley, A. D., & Logie, R. H. (1999). Working memory: The multiple-component model. In A. Miyake & P. Shah (Eds). Models of working memory: Mechanisms of active maintenance and executive control (pp. 28-61). Cambridge, England: Cambridge University Press.

Baker, D., Prince, C., Shrestha, L., Oser, R., & Salas, E. (1993). Aviation computer games for crew resource management training. The International Journal of Aviation Psychology, 3(2), 143-156.

Baker, E. L., & Mayer, R. E. (1999). Computer-based assessment of problem solving. Computers in Human Behavior, 15, 269-282.

Baker, E. L. & O’Neil, H. F., Jr. (2002). Measuring problem solving in computer environments: current and future states. Computers in Human Behavior, 18, 609-622.

Banbury, S. P., Macken, W. J., Tremblay, S., & Jones, D. M. (2001, Spring). Auditory distraction and short-term memory: Phenomena and practical implications. Human Factors, 43(1), 12-29.

Bandura, A. (1997). Self-efficacy: The exercise of control. New York, NY: Freeman.

Barab, S. A., Bowdish, B. E., & Lawless, K. A. (1997). Hypermedia navigation: Profiles of hypermedia users. Educational Technology Research & Development, 45(3), 23-41.

Barab, S. A., Young, M. F., & Wang, J. (1999). The effects of navigational and generative activities in hypertext learning on problem solving and comprehension. International Journal of Instructional Media, 26(3), 283-309.

Baylor, A. L. (2001). Perceived disorientation and incidental learning in a web-based environment: Internal and external factors. Journal of Educational Multimedia and Hypermedia, 10(3), 227-251.

Benbasat, I., & Todd, P. (1993). An experimental investigation of interface design alternatives: Icon vs. text and direct manipulation vs. menus. International Journal of Man-Machine Studies, 38, 369-402.

241


Berson, M. J. (1996, Summer). Effectiveness of computer technology in the social studies: A review of the literature. Journal of Research on Computing in Education, 28(4), 486-499.

Berube, M. S., Severynse, M., Jost, D. A., Ellis, K., Pickett, J. P., Previte, R. E., et al. (Eds.) (2001). Webster’s II: New college dictionary. Boston, MA: Houghton Mifflin Company.

Berylne, D. E. (1960). Conflict, arousal, and curiosity. New York: McGraw-Hill.

Betz, J. A. (1995/1996). Computer games: Increase learning in an interactive multidisciplinary environment. Journal of Educational Technology Systems, 24(2), 195-205.

Bong, M. (2001). Between- and within-domain relationships of academic motivation among middle and high school students: Self-efficacy, task-value, and achievement goals. Journal of Educational Psychology, 93(1), 23-34.

Borkowski, J. G., Pintrich, P. R., Zeidner, M. H. (Eds). (2000). Handbook of Self-Regulation. San Diego, CA: Academic

Brougere, G. (1999, June). Some elements relating to children’s play and adult simulation/gaming. Simulation & Gaming, 30(2), 134-146.

Brown, D. W., & Schneider, S. D. (1992), Young learners’ reactions to problem solving contrasted by distinctly divergent computer interfaces. Journal of Computing in Childhood Education, 3(3/4), 335-347.

Brozik, D., & Zapalska, A. (2002, June). The PORTFOLIO GAME: Decision making in a dynamic environment. Simulation & Gaming, 33(2), 242-255.

Brunken, R., Plass, J. L., & Leutner, D. (2003). Direct measurement of cognitive load in multimedia learning. Educational Psychologist 38(1), 53-61.

Brunning, R. H., Schraw, G. J., & Ronning, R R. (1999). Cognitive psychology and instruction (3rd ed.). Upper Saddle River, NJ: Merrill.

Carr, P. D., & Groves, G. (1998). The Internet-based operations simulation game. In J. A. Chambers (Ed.), Selected Papers for the 9th International Conference on college Teaching and Learning (pp. 15-23). Jacksonville, FL, US: Florida Community Collage at Jacksonville.

Carroll, W. M. (1994). Using worked examples as an instructional support in the algebra classroom. Journal of Educational Psychology, 86(3), 360-367.

Chalmers, P. A. (2003). The role of cognitive theory in human-computer interface. Computers in Human Behavior, 19, 593-607.

242


Chen, H. H. (2005) A formative evaluation of the training effectiveness of a computer game. Unpublished doctoral dissertation. University of Southern California.

Chi, M. T. H., Bassok, M., Lewis, M. W., Reimann, P., & Glaser, R. (1989). Self-explanations: How students study and use examples in learning to solve problems. Cognitive Science, 13, 145-182.

Chou, C., & Lin, H. (1998). The effect of navigation map types and cognitive styles on learners’ performance in a computer-networked hypertext learning system [Electronic Version]. Journal of Educational Multimedia and Hypermedia, 7(2/3), 151-176.

Chou, C., Lin, H, & Sun, C.-t. (2000). Navigation maps in hierarchical-structured hypertext courseware [Electronic Version]. International Journal of Instructional Media, 27(2), 165-182.

Chuang, S.-h. (2003). The role of search strategies and feedback on a computer-based problem solving task. Unpublished doctoral dissertation. University of Southern California.

Clark, R. E. (1999). The CANE model of motivation to learn and to work: A two-stage process of goal commitment and effort [Electronic Version]. In J. Lowyck (Ed.), Trends in Corporate Training. Leuven, Belgium: University of Leuven Press.

Clark, R. E. (Ed.) (2001). Learning from Media: Arguments, analysis, and evidence. Greenwich, CT: Information Age Publishing.

Clark, R. E. (2003, February). Strategies based on effective feedback during learning. In H. F. O’Neil (Ed.), What Works in Distance Education. Report to the Office of Naval Research by the National Center for Research on Evaluation, Standards, and Student Testing (pp. 18-19).

Clark, R. E. (2003b, February). Strategies based on increasing student motivation: Encouraging active engagement and persistence. In H. F. O’Neil (Ed.), What Works in Distance Education. Report to the Office of Naval Research by the National Center for Research on Evaluation, Standards, and Student Testing (pp. 20-21).

Clark, R. E. (2003b, February). Strategies based on providing learner control of instructional navigation. In H. F. O’Neil (Ed.), What Works in Distance Education. Report to the Office of Naval Research by the National Center for Research on Evaluation, Standards, and Student Testing (pp. 14-15).

Clark, R. E. (2003d, March). Fostering the work motivation of teams and individuals. Performance Improvement, 42(3), 21-29.

243


Clark, R. E., Sugrue, B. M. (2001). International views of the media debate. In R. E. Clark (Ed.), Learning from Media: Arguments, Analysis, and Evidence. (pp. 71-88. Greenwich, CT: Information Age Publishing

Cobb, T. (1997). Cognitive efficiency: Toward a revised theory of media. Educational Technology Research and Development, 45(4), 21-35.

Coffin, R. J., & MacIntyre, P. D. (1999). Motivational influences on computer-related affective states. Computers in Human Behavior, 15, 549-569.

Corno, L., & Mandinach, E. B. (1983). The role of cognitive engagement in classroom learning and motivation. Educational Psychologist, 18(2), 88-108.

Crookall, D., & Aria, K. (Eds.). (1995). Simulation and gaming across disciplines and cultures: ISAGA at a watershed. Thousand Oaks, CA: Sage.

Crookall, D., Oxford, R. L., & Saunders, D. (1987). Towards a reconceptualization of simulation. From representation to reality. Simulation/Games for Learning, 17, 147-171.

Cross, T. L. (1993, Fall). AgVenture: A farming strategy computer game. Journal of Natural Resources and Life Sciences Education, 22, 103-107.

Csikszentmihalyi, M. (1975). Beyond boredom and anxiety. San Francisco: Jossey Bass.

Csikszentmihalyi, M. (1990). Flow: The psychology of optimal performance. New York: Cambridge University Press.

Cutmore, T. R. H., Hine, T. J., Maberly, K. J., Langford, N. M., & Hawgood, G. (2000). Cognitive and gender factors influencing navigation in a virtual environment. International Journal of Human-Computer Studies, 53, 223-249.

Daniels, H. L., & Moore, D. M. (2000). Interaction of cognitive style and learner control in a hypermedia environment. International Journal of Instructional Media, 27(4), 369-383.

Davis, S., & Wiedenbeck, S. (2001). The mediating effects of intrinsic motivation, ease of use and usefulness perceptions on performance in first-time and subsequent computer users. Interacting with Computers, 13, 549-580.

Day, E. A., Arthur, W., Jr., & Gettman, D. (2001). Knowledge structures and the acquisition of a complex skill. Journal of Applied Psychology, 86(5), 1022-1033.

244


de Jong, T., de Hoog, R., & de Vries, F. (1993). Coping with complex environments: The effects of providing overviews and a transparent interface on learning with a computer simulation. International Journal of Man-Machine Studies, 39, 621-639.

de Jong, T., & van Joolingen, W. R. (1998). Scientific discovery learning with computer simulations of conceptual domains. Review of Educational Research, 68, 179-202.

deCharms, R. (1986). Personal Causation. New York: Academic Press.

Deci, E. L. (1975). Intrinsic Motivation. New York: Plenum Press.

Dekkers, J., & Donati, S. (1981). The interpretation of research studies on the use of simulation as an instructional strategy. Journal of Educational Research, 74(6), 64-79.

Dempsey, J. V., Haynes, L. L., Lucassen, B. A., & Casey, M. S. (2002). Forty simple computer games and what they could mean to educators. Simulation & Gaming, 43(2), 157-168.

Dias, P., Gomes, M. J., & Correia, A. P. (1999). Disorientation in hypermedia environments: Mechanisms to support navigation. Journal of Educational Computing Research, 20(2), 93-117.

Dillon, A., & Gabbard, R. (1998, Fall). Hypermedia as an educational technology: A review of the quantitative research literature on learner comprehension, control, and style. Review of Educational Research, 63(3), 322-349.

Donchin, E. (1989). The learning strategies project. Acta Psychologica, 71, 1-15.

Druckman, D. (1995). The educational effectiveness of interactive games. In D. Crookall & K. Aria (Eds.), Simulation and gaming across disciplines and cultures: ISAGA at a watershed (pp. 178-187). Thousand Oaks, CA: Sage

Eberts, R. E., & Bittianda, K. P. (1993). Preferred mental models for direct-manipulation and command-based interfaces. International Journal of Man-Machine Studies, 38, 769-785.

Eccles, J. S., & Wigfeld, A. (2002). Motivational beliefs, values, and goals. Annual Review of Psychology, 53, 109-132.

Farrell, I. H., & Moore, D. M. (2000). The effect of navigation tools on learners’ achievement and attitude in a hypermedia environment. Journal of Educational Technology Systems, 29(2), 169-181.

245


Frohlich, D. M. (1997). Direct manipulation and other lessons. In M. Helander, T. K. Landauer & P. Prabhu (eds.), Handbook of Human Computer Interaction: Second, Completely Revised Edition (pp. 463-488). Amsterdam: Elsevier

Galimberti, C., Ignazi, S., Vercesi, P., & Riva, G. (2001). Communication and cooperation in networked environment: An experimental analysis. CyberPsychology & Behavior, 4(1), 131-146.

Garris, R., Ahlers, R., & Driskell, J. E. (2002). Games, motivation, and learning: A research and practice model. Simulation & Gaming, 33(4), 441-467.

Gevins, A., Smith, M. E., Leong, H., McEvoy, L., Whitfield, S., Du, R., & Rush, G. (1998). Monitoring working memory load during computer-based tasks with EEG pattern recognition methods. Human Factors, 40(1), 79-91.

Gopher, D., Weil, M., & Bareket, T. (1994). Transfer of skill from a computer game trainer to flight. Human Factors, 36(3), 387-405.

Gredler, M.E. (1996). Educational games and simulations: a technology in search of a research paradigm. In D. H. Jonassen (Ed.). Handbook of Research for Educational Communications and Technology. (pp 521-540). New York: Simon & Schuster Macmillan.

Green, C. S., & Bavelier, D. (2003, May 29). Action video game modifies visual selective attention. Nature, 423, 534-537.

Green, T. D., & Flowers, J. H. (2003). Comparison of implicit and explicit learning processes in a probabilistic task. Perceptual and Motor Skills, 97, 299-314.

Greenfield, P. M., deWinstanley, P., Kilpatrick, H., & Kaye, D. (1996). Action video games and informal education: Effects on strategies for dividing visual attention. In P. M. Greenfield & R. R. Cocking (Eds.), Interacting with Video (pp. 187-205). Norwood, NJ: Ablex Publishing Corporation.

Hannifin, R. D., & Sullivan, H. J. (1996). Preferences and learner control over amount of instruction. Journal of Educational Psychology, 88, 162-173.

Harp, S. F., & Mayer, R. E. (1998). How seductive details do their damage: A theory of cognitive interest in science learning. Journal of Educational Psychology, 90(3), 414-434.

Harter, S. (1978). Effectance motivation reconsidered: Toward a developmental model. Human Development, 1, 34-64.

Henderson, L., Klemes, J., & Eshet, Y. (2000). Just playing a game? Educational simulation software and cognitive outcomes. Journal of Educational Computing Research, 22(1), 105-129.

246


Herl, H. E., Baker, E. L., & Niemi, D. (1996). Construct validation of an approach to modeling cognitive structure of U.S. history knowledge. Journal of Educational Psychology, 89(4), 206-218.

Herl, H. E., O’Neil, H. F., Jr., Chung, G., & Schacter, J. (1999) Reliability and validity of a computer-based knowledge mapping system to measure content understanding. Computer in Human Behavior, 15, 315-333.

Hong, E., & O’Neil, H. F. Jr. (2001). Construct validation of a trait self-regulation model. International Journal of Psychology, 36(3), 186-194.

Howland, J., Laffey, J., & Espinosa, L. M. (1997). A computing experience to motivate children to complex performances [Electronic Version]. Journal of Computing in Childhood Education, 8(4), 291-311.

Hubbard, P. (1991, June). Evaluating computer games for language learning. Simulation & Gaming, 22(2), 220-223.

Jones, M. G., Farquhar, J. D., & Surry, D. W. (1995, July/August). Using metacognitive theories to design user interfaces for computer-based learning. Educational Technology, 35(4), 12-22.

Kaber, D. B., Riley, J. M., & Tan, K.-W. (2002). Improved usability of aviation automation through direct manipulation and graphical user interface design. The International Journal of Aviation Psychology, 12(2), 153-178.

Kagan, J. (1972). Motives and development. Journal of Personality and Social Psychology, 22, 51-66.

Kalyuga, S., Ayres, P., Chandler, P., & Sweller, J. (2003). The expertise reversal effect. Educational Psychologist, 38(1), 23-31.

Kalyuga, S., Chandler, P., & Sweller, J. (1998). Levels of expertise and instructional design. Human Factors, 40(1), 1-17.

Kee, D. W., & Davies, L. (1988). Mental effort and elaboration: A developmental analysis. Contemporary Educational Psychology, 13, 221-228.

Kee, D. W., & Davies, L. (1990). Mental effort and elaboration: Effects of Accessibility and instruction. Journal of Experimental Child Psychology, 49, 264-274.

Khoo, G.-s., & Koh, t.-s. (1998). Using visualization and simulation tools in tertiary science education [Electronic Version]. The Journal of Computers in Mathematics and Science Teaching, 17(1), 5-20.

247


King, K. W., & Morrison, M. (1998, Autumn). A media buying simulation game using the Internet. Journalism and Mass Communication Educator, 53(3), 28-36.

Kirriemuir, J. (2002). The relevance of video games and gaming consoles to the higher and further education learning experience. Retrieved 2/3/2004 from http://www.jisc.ac.uk/general/index.cfm?name=techwatch_report_0201

Kirriemuir, J. (2002b, February). Video gaming, education, and digital learning technologies: Relevance and opportunities [Electronic Version]. D-Lib Magazine, 8(2), 1-8.

Lee, J. (1999). Effectiveness of computer-based instructional simulation: A meta analysis. International Journal of Instructional Media, 26(1), 71-85.

Leemkuil, H., de Jong, T., de Hoog, R., & Christoph, N. (2003). KM Quest: A collaborative Internet-based simulation game. Simulation & Gaming, 34(1), 89-111.

Locke, E. A., & Latham, G. P. (1990). A theory of goal setting and task performance. Englewood Cliffs, NJ: Prentice Hall.

Locke, E. A., Latham, G. P. (2002, Summer). Building a practically useful theory of goal setting and task motivation: A 35-year odyssey. American Psychologist, 57(9), 705-717.

Malone, T. W. (1981). What makes computer games fun? Byte, 6(12), 258-277.

Malone, T. W., & Lepper, M. R. (1987). Making leraning fun: A taxonomy of intrinsic motivation for learning. In R. E. Snow & M. J. Farr (Eds.). Aptitute, learning, and instruction: Vol. 3. Conative and affective process analyses (pp. 223-253). Hillsdale, NJ: Lawrence Erlbaum.

Malouf, D. (1987-1988). The effect of instructional computer games on continuing student motivation. The Journal of Special Education, 21(4), 27-38.

Mayer, R. E. (1981). A psychology of how novices learn computer programming. Computing Surveys, 13, 121-141.

Mayer, R. E. (1998). Cognitive, metacognitive, and motivational aspects of problem solving. Instructional Science, 26, 49-63.

Mayer, R. E. (2001). Multimedia Learning. Cambridge, UK: Cambridge University Press.

Mayer, R. E. (2003). Learning and Instruction. Upper Saddle Ridge, NJ: Pearson Education.

248


Mayer, R. E., & Chandler, P. (2001). When learning is just a click away: Does simple user interaction foster deeper understanding of multimedia messages? Journal of Educational Psychology, 93(2), 390-397.

Mayer, R. E., Heiser, J., & Lonn, S. (2001). Cognitive constraints on multimedia learning: When presenting more material results in less understanding. Journal of Educational Psychology, 93(1), 187-198.

Mayer, R. E., Mautone, P., & Prothero, W. (2002). Pictorial aids for learning by doing in a multimedia geology simulation game. Journal of Educational Psychology, 94(1), 171-185.

Mayer, R. E., & Moreno, R. (1998). A split-attention effect in multimedia learning: Evidence of dual processing systems in working memory. Journal of Educational Psychology, 90(2), 312-320.

Mayer, R. E., & Moreno, R. (2003). Nine ways to reduce cognitive load in multimedia learning. Educational Psychologist, 38(1), 43-52.

Mayer, R. E., Moreno, R., Boire, M., & Vagge, S. (1999). Maximizing constructivist learning from multimedia communications by minimizing cognitive load. Journal of Educational Psychology, 91(4), 638-643.

Mayer, R. E., Sobko, K., & Mautone, P. D. (2003). Social cues in multimedia learning: Role of speaker’s voice. Journal of Educational Psychology, 95(20), 419-425.

Mayer, R. E., & Wittrock, M. C. (1996). Problem solving transfer. In D. C. Berliner & R. C. Calfee (Eds.), Handbook of educational psychology (pp. 47-62). New York: Simon & Schuster Macmillan.

McGrenere, J. (1996). Design: Educational electronic multi-player games—A literature review (Technical Report No. 96-12, the University of British Columbia). Retrieved from http://taz.cs.ubc.ca/egems/papers/desmugs.pdf

Miller, G. A. (1956). The magical number, seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63, 81-97.

Moreno, R., & Mayer, R. E. (1999). Cognitive principles of multimedia learning: The role of modality and contiguity. Journal of Educational Psychology, 91(2), 358-368.

Moreno, R., & Mayer, R. E. (2000a). A coherence effect in multimedia learning: The case of minimizing irrelevant sounds in the design of multimedia instructional messages. Journal of Educational Psychology, 92(1), 117-125.

249


Moreno, R., & Mayer, R. E. (2000b). Engaging students in active learning: The case for personalized multimedia messages. Journal of Educational Psychology, 92(4), 724-733.

Moreno, R., & Mayer, R. E. (2002). Learning science in virtual reality multimedia environments: Role of methods and media. Journal of Educational Psychology, 94(3), 598-610.

Morris, C. S., Hancock, P. A., & Shirkey, E. C. (2004). Motivational effects of adding context relevant stress in PC-based game training. Military Psychology, 16, 135-147.

Mousavi, S. Y., Low, R., & Sweller, J. (1995). Reducing cognitive load by mixing auditory and visual presentation modes. Journal of Educational Psychology, 87(2), 319-334.

Mwangi, W., & Sweller, J. (1998). Learning to solve compare word problems: The effect of example format and generating self-explanations. Cognition and Instruction, 16(2), 173-199.

Niemiec, R. P., Sikorski, C., & Walberg, H. J. (1996). Learner-control effects: A review of reviews and a meta-analysis. Journal of Educational Computing Research, 15(2), 157-174.

Novak, J. (2005). Game Development Essentials. Clifton Park, NY: Thomson Delmar Learning.

Noyes, J. M., & Garland, K. J. (2003). Solving the Tower of Hanoi: Does mode of presentation matter? Computers in Human Behavior, 19, 579-592.

O’Neil, H. F., Jr. (1999). Perspectives on computer-based performance assessment of problem solving: Editor’s introduction. Computers in Human Behavior, 15, 255-268.

O'Neil, H. F., Jr. (2002). Perspective on computer-based assessment of problem solving [Special Issue]. Computers in Human Behavior, 18(6), 605-607.

O’Neil, H. F., Jr., & Abedi, J. (1996). Reliability and validity of a state metacognitive inventory: Potential for alternative assessment. Journal of Educational Research, 89, 234-245.

O’Neil, H. F., Jr., Baker, E. L., Fisher, J. Y.-C. (2002, August 31). A formative evaluation of ICT games. Manuscript: University of California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing (CRESST)

250


O’Neil, H. F., Jr., & Herl, H. E. (1998). Reliability and validity of a trait measure of self-regulation. Manuscript: University of California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).

O’Neil, H. F., & Wainess, R. (in preparation). Classification of learning outcomes: Evidence from the computer games literature. The Curriculum Journal.

Paas, F., Renkl, A., & Sweller, J. (2003). Cognitive load theory and instructional design: Recent developments. Educational Psychologist, 38(1), 1-4.

Paas, F., Tuovinen, J. E., Tabbers, H., & Van Gerven, P. W. M. (2003). Cognitive load measurement as a means to advance cognitive load theory. Educational Psychologist, 38(1), 63-71.

Parchman, S. W., Ellis, J. A., Christinaz, D., & Vogel, M. (2000). An evaluation of three computer-based instructional strategies in basic electricity and electronics training. Military Psychology, 12(1), 73-87.

Park, O.-C., & Gittelman, S. S. (1995). Dynamic characteristics of mental models and dynamic visual displays. Instructional Science, 23, 303-320.

Perkins, D. N., & Salomon, G. (1989). Are cognitive skills context bound? Educational Researcher, 18, 16-25.

Pintrich, P. R., & DeGroot, E. V. (1990). Motivational and self-regulated learning components of classroom academic performance. Journal of Educational Psychology, 82, 33-40.

Pintrich, P. R., & Schunk, D. H. (2002). Motivation in education: Theory, research, and applications. Upper Saddle River, NJ: Pearson Education.

Plotnick, E. (1999). Concept mapping: A graphical system for understanding the relationship between concepts. Educational Media & Technology Yearbook, 24, 81-84.

Porter, D. B., Bird, M. E., & Wunder, A. (1990-1991). Competition, cooperation, satisfaction, and the performance of complex tasks among Air Force cadets. Current Psychology: Research & Reviews, 9(4), 347-354.

Prislin, R., Jordan, J. A., Worchel, S., Semmer, F. T., & Shebilske, W. L. (1996, September). Effects of group discussion on acquisition of complex skills. Human Factors, 38(3), 404-416.

Quilici, J. L., & Mayer, R. E. (1996). Role of examples in how students learn to categorize statistics word problems. Journal of Educational Psychology, 88(1), 144-161.

251


Ramsberger, P. F., Hopwood, D., Hargan, C. S., & Underhill, W. G. (1983), Evaluation of a spatial data management system for basic skills education. Final Phase I Report for Period 7 October 1980 - 30 April 1983 (HumRRO FR-PRD-83-23). Alexandria, VA: Human Resources Research Organization.

Randel, J. M., Morris, B. A., Wetzel, C. D., & Whitehill, B. V. (1992). The effectiveness of games for educational purposes: A review of recent research. Simulation & Games, 23, 261-276.

Renkl, A., & Atkinson, R. K. (2003). Structuring the transition from example study to problem solving in cognitive skill acquisition: A cognitive load perspective. Educational Psychologist, 38(1), 13-22.

Renkl, A., Atkinson, R. K., Maier, U. H., & Staley, R. (2002). From example study to problem solving: Smooth transitions help learning. The Journal of Experimental Education, 70(4), 293-315.

Resnick, H., & Sherer, M. (1994). Computerized games in the human services--An introduction. In H. Resnick (Ed.), Electronic Tools for Social Work Practice and Education (pp. 5-16). Bington, NY: The Haworth Press.

Rhodenizer, L. , Bowers, C. A., & Bergondy, M. (1998). Team practice schedules: What do we know? Perceptual and Motor Skills, 87, 31-34.

Ricci, K. E. (1994, Summer). The use of computer-based videogames in knowledge acquisition and retention. Journal of Interactive Instruction Development, 7(1), 17-22.

Ricci, K. E., Salas, E., & Cannon-Bowers, J. A. (1996). Do computer-based games facilitate knowledge acquisition and retention? Military Psychology, 8(4), 295-307.

Rieber, L. P. (1996). Seriously considering play: Designing interactive learning environments based on the blending of microworlds, simulations, and games. Educational Technology Research and Development, 44(2), 43-58.

Rieber, L. P., & Matzko, M. J. (Jan/Feb 2001). Serious design for serious play in physics. Educational Technology, 41(1), 14-24.

Rieber, L. P., Smith, L., & Noah, D. (1998, November/December). The value of serious play. Educational Technology, 38(6), 29-37.

Rosenorn, T., & Kofoed, L. B. (1998). Reflection in learning processes through simulation/gaming. Simulation & Gaming, 29(4), 432-440.

252


Ruben, B. D. (1999, December). Simulations, games, and experience-based learning: The quest for a new paradigm for teaching and learning. Simulations & Gaming, 30, 4, 498-505.

Ruddle, R. A., Howes, A., Payne, S. J., & Jones, D. M. (2000). The effects of hyperlinks on navigation in virtual environments. International Journal of Human-Computer Studies, 53, 551-581.

Ruiz-Primo, M. A., Schultz, S. E., and Shavelson, R. J. (1997). Knowledge map-based assessment in science: Two exploratory studies (CSE Tech. Rep. No. 436). Los Angeles, University if California, Center for Research on Evaluation, Standards, and Student Testing (CRESST).

Salas, E., Bowers, C. A., & Rhodenizer, L. (1998). It is not how much you have but how you use it: Toward a rational use of simulation to support aviation training. The International Journal of Aviation Psychology, 8(3), 197-208.

Salomon, G. (1983). The differential investment of mental effort in learning from different sources. Educational Psychology, 18(1), 42-50.

Santos, J. (2002, Winter). Developing and implementing an Internet-based financial system simulation game. The Journal of Economic Education, 33(1), 31-40.

Schau, C. & Mattern, N. (1997). Use of map techniques in teaching applied statistics courses. American statistician, 51, 171-175.

Schraw, G. (1998). Processing and recall differences among seductive details. Journal of Educational Psychology, 90(1), 3-12.

Shen, C.-Y. (in preparation). The Effectiveness of Worked Examples in a Game-Based Problem-Solving Task. Unpublished doctoral dissertation. University of Southern California.

Shebilske, W. L., Regian, W., Arthur, W., Jr., & Jordan, J. A. (1992). A dyadic protocol for training complex skills. Human Factors, 34(3), 369-374.

Shewokis, P. A. (2004). Memory consolidation and contextual interference effects with computer games. Perceptual and Motor Skills, 97, 381-389.

Shyu, H.-y., & Brown, S. W. (1995). Learner-control: The effects of learning a procedural task during computer-based videodisc instruction. International Journal of Instructional Media, 22(3), 217-230.

Soanes, C. (Ed.) (2003). Compact Oxford English Dictionary of Current English (second edition). Oxford, UK: Oxford University Press

253


Spiker, V. A., & Nullmeyer, R. T. (n.d.). Benefits and limitations of simulation-based mission planning and rehearsal. Unpublished manuscript.

Stewart, K. M. (1997, Spring). Beyond entertainment: Using interactive games in web-based instruction. Journal of Instructional Delivery, 11(2), 18-20.

Stolk, D., Alexandrian, D., Gros, B., & Paggio, R. (2001). Gaming and multimedia applications for environmental crisis management training. Computers in Human Behavior, 17, 627-642.

Story, N., & Sullivan, H. J. (1986, November/December). Factors that influence continuing motivation. Journal of Educational Research, 80(2), 86-92.

Sweller, J., & Chandler, P. (1994). Why some material is difficult to learn. Cognition and Instruction, 12, 185-233.

Tarmizi, R. A., & Sweller, J. (1988). Guidance during mathematical problem solving. Journal of Educational Psychology, 80(4), 424-436.

Tennyson, r. D., & Breuer, K. (2002). Improving problem solving and creativity through use of complex-dynamic simulations. Computers in Human Behavior, 18(6), 650-668.

Thiagarajan, S. (1998, Sept/October). The myths and realities of simulations in performance technology. Educational Technology, 38(4), 35-41.

Thomas, P., & Macredie, R. (1994). Games and the design of human-computer interfaces. Educational Technology, 31, 134-142.

Thompson, L. F., Meriac, J. P., & Cope, J. G. (2002, Summer). Motivating online performance: The influence of goal setting and Internet self-efficacy. Social Science Computer Review, 20(2), 149-160.

Thorndyke, P. W., & Hayes-Roth, B. (1982). Differences in spatial knowledge acquired from maps and navigation. Cognitive Psychology, 14, 560-589.

Tkacz, S. (1998, September). Learning map interpretation: Skill acquisition and underlying abilities. Journal of Environmental Psychology, 18(3), 237-249.

Tuovinen, J. E., & Sweller, J. (1999). A comparison of cognitive load associated with discovery learning and worked examples. Journal of Educational Psychology, 91(2), 334-341.

van Merrienboer, J. J. G., Clark, R. E., & de Croock, M. B. M. (2002). Blueprints for complex learning: The 4C/ID-model. Educational Technology Research & Development, 50(2), 39-64.

254


van Merrienboer, J. J. G., Kirschner, P. A., & Kester, L. (2003). Taking a load off a learner’s mind: Instructional design for complex learning. Educational Psychologist, 38(1), 5-13.

Wainess, R., & O’Neil, H. F. Jr. (2003, August). Feasibility study: Video game research platform. Manuscript: University of Southern California.

Washbush, J., & Gosen, J. (2001, September). An exploration of game-derived learning in total enterprise simulations. Simulation & Gaming, 32(3), 281-296.

West, D. C., Pomeroy, J. R., Park, J. K., Gerstenberger, E. A., Sandoval, J. (2000). Critical thinking in graduate medical education. Journal of the American Medical Association, 284(9), 1105-1110.

Westbrook, J. I., & Braithwaite, J. (2001). The Health Care Game: An evaluation of a heuristic, web-based simulation. Journal of Interactive Learning Research, 12(1), 89-104.

Westerman, S. J. (1997). Individual differences in the use of command line and menu computer interfaces. International Journal of Human-Computer Interaction, 9(2), 183-198.

White, R. W. (1959). Motivation reconsidered: The concept of competence. Psychological Review, 66, 297-333.

Wiedenbeck, S., & Davis, S. (1997). The influence of interaction style and experience on user perceptions of software packages. International Journal of Human-Computer Studies, 46, 563-588.

Wolfe, J. (1997, December). The effectiveness of business games in strategic management course work [Electronic Version]. Simulation & Gaming Special Issue: Teaching Strategic Management, 28(4), 360-376.

Wolfe, J., & Roge, J. N. (1997, December). Computerized general management games as strategic management learning environments [Electronic Version]. Simulation & Gaming Special Issue: Teaching Strategic Management, 28(4), 423-441.

Yair, Y., Mintz, R., & Litvak, S. (2001). 3D-virtual reality in science education: An implication for astronomy teaching. Journal of Computers in Mathematics and Science Teaching, 20(3), 293-305.

Yeung, A. S., Jin, P., & Sweller, J. (1997). Cognitive load and learner expertise: Split-attention and redundancy effects in reading with explanatory notes. Contemporary Educational Psychology, 23, 1-21.

255


Yu, F.-Y. (2001). Competition within computer-assisted cooperative learning environments: Cognitive, affective, and social outcomes. Journal of Educational Computing Research, 24(2), 99-117.

Yung, A. S. (1999). Cognitive load and learner expertise: Split attention and redundancy effects in reading comprehension tasks with vocabulary definitions. Journal of Educational Media, 24(2), 87-102.

Zimmerman, B. J. (1994). Dimensions of academic self-regulation: A conceptual framework for education. In D. H. Schunk, & B. J. Zimmerman (Eds.), Self-regulation of learning and performance (pp. 3-21). Hillsdale, NJ: Erlbaum.

Zimmerman, B. J. (2000). Self-efficacy. An essential motive to learn. Contemporary Educational Psychology, 25(1), 82-91.

256


APPENDIX A

Self-Regulation Questionnaire

Name (please print): __________________________________________________

Directions: A number of statements which people have used to describe themselves are given below. Read each statement and indicate how you generally think or feel on learning tasks by marking your answer sheet. There are no right or wrong answers. Do not spend too much time on any one statement. Remember, give the answer that seems to describe how you generally think or feel.

AlmostNever Sometimes Often

AlmostAlways

1. I determine how to solve a task before I begin.

1 2 3 4

2. I check how well I am doing when I solve a task.

1 2 3 4

3. I work hard to do well even if I don't like a task.

1 2 3 4

4. I believe I will receive an excellent grade in courses.

1 2 3 4

5. I carefully plan my course of action.

1 2 3 4

6. I ask myself questions to stay on track as I do a task.

1 2 3 4

7. I put forth my best effort on tasks.

1 2 3 4

8. I’m certain I can understand the most difficult material presented in the readings for courses.

1 2 3 4

9. I try to understand tasks before I attempt to solve them.

1 2 3 4

10. I check my work while I am doing it.

1 2 3 4

11. I work as hard as possible on tasks.

1 2 3 4

257



AlmostAlways

12. I’m confident I can understand the basic concepts taught in courses.

1 2 3 4

13. I try to understand the goal of a task before I attempt to answer.

1 2 3 4

14. I almost always know how much of a task I have to complete.

1 2 3 4

15. I am willing to do extra work on tasks to improve my knowledge.

1 2 3 4

16. I’m confident I can understand the most complex material presented by the teacher in courses.

1 2 3 4

17. I figure out my goals and what I need to do to accomplish them.

1 2 3 4

18. I judge the correctness of my work.

1 2 3 4

19. I concentrate as hard as I can when doing a task.

1 2 3 4

20. I’m confident I can do an excellent job on the assignments and tests in courses.

1 2 3 4

21. I imagine the parts of a task I have to complete.

1 2 3 4

22. I correct my errors. 1 2 3 4

23. I work hard on a task even if it does not count.

1 2 3 4

24. I expect to do well in this course. 1 2 3 4

25. I make sure I understand just what has to be done and how to do it.

1 2 3 4

26. I check my accuracy as I progress through a task.

1 2 3 4

258



AlmostAlways

27. A task is useful to check my knowledge.

1 2 3 4

28. I’m certain I can master the skills being taught in courses.

1 2 3 4

29. I try to determine what the task requires.

1 2 3 4

30. I ask myself, how well am I doing, as I proceed through tasks.

1 2 3 4

31. Practice makes perfect. 1 2 3 4

32. Considering the difficulty of courses, teachers, and my skills, I think I will do well courses.

1 2 3 4

Copyright © 1995, 1997, 1998, 2000 by Harold F. O’Neil, Jr.

259


APPENDIX B

Knowledge Map Specifications

General Domain Specification

This Software

Scenario Create a knowledge map of the content understanding of SafeCracker, a computer puzzle-solving game.

Participants College students, graduates, or graduate students.Knowledge map

concepts/nodesFifteen predefined key concepts identified in the content of Safecracker by multiple experts; book, catalog, clue, code, combination, compass, desk, direction, floor plan, key, room, safe, searching, trial-and-error, and tool.

Knowledge map links

Seven predefined relational links identified in the content of SafeCracker by multiple experts: causes, contains, leads to, part of, prior to, requires, and used for.

Knowledge map domain/content: SafeCracker

SafeCracker is a computer puzzle-solving game. There are over 50 rooms containing approximately 30 safes; each safe is a puzzle to solve. Five rooms were used in the study—three for each game, with one room used in both games. To solve the puzzles, participants must find clues and tools hidden in the rooms, deliberate and reason the logic and sequence of a safe, and attempt to apply items and clues they have found. In some instances, participants must also apply prior domain knowledge.

Training of the computer knowledge mapping system

All participants went through the same training session with one exception; those in the treatment group learned to use the navigation map and the treatment and control groups were given different path finding strategies.The training included the following elements:• How to construct a knowledge map using the computer mapping system• How to play SafeCracker

Type of knowledge to be learned

Problem solving

Three problem solving measures

1. Knowledge map used to measure content understanding and structure, including (a) semantic content score; (b) the number of concepts; and (c) the number of links

2. Domain specific problem solving strategy questionnaire, including questions to measure problem solving retention and transfer

3. Trait self-regulation questionnaire used to measures the four elements of trait self-regulation: planning, self-checking, self-efficacy, and mental effort

260

Engineering Class Home Pages - A Formative Evaluation of ... · Web view2005/11/13 · Webster et...

Documents

Transcript of Engineering Class Home Pages - A Formative Evaluation of ... · Web view2005/11/13 · Webster et...