Commercial Video Games as Preparation for Future Learning

Commercial Video Games

as Prepara1on for Future Learning

Dylan Arena

Stanford University

June 2012

Tuesday, June 26, 12

Hi! OK, I’m gonna get started. I’ll be talking today about the poten1al of using commercial video games as prepara1on for future learning.

Background


First I’ll set the stage a little bit…


We in the GLS community claim that gameplay can be great for learning, and by that we tend to mean two things:


First, that we can design learning-‐based games (like Quest Atlan1s) to support “schoolish” learning…


…and second, that we can observe other interes1ng kinds of learning even in commercial, off-‐the-‐shelf games (like World of WarcraY).


A third claim—that simply playing commercial games recrea1onally…


…could help with…


…schoolish stuff—


—seems a bit silly at first, if only because most schoolish tests focus on retrieval of facts that most commercial games aren’t designed to teach.


But if we broaden our view, it’s a bit less silly.

Theory


Here’s a thought experiment to help make clear why.


Imagine spli]ng a class in half: half get to explore a forest (turning over rocks, looking inside ro_en logs, poking ant hills) and half don’t (they just stand quietly facing the wall).

Forest ecosystem test


Then bring both groups back into class and give ‘em a tradi1onal mul1ple-‐choice test about forest ecosystems. You’d expect no difference, right? It’s unlikely that just exploring the forest would give kids the kind of knowledge they’d need to pick correct answers on a mul1ple-‐choice test.


Forest Ecosystem Lecture


But imagine then giving a LECTURE about forest ecosystems—ground cover, canopy, decay and new growth, etc. It’s possible that the kids who had just been out exploring a forest might engage more with that lecture and hence learn more.


Forest Ecosystem Lecture



If you then test everyone again aYer the lecture, you might observe some previously hidden benefits of the forest field trip.

The point is that what we bring into a learning situa1on (like a lecture) is obviously very important. But it can be hard to measure what we bring in, especially when it isn’t stable, well-‐structured factual knowledge.

Prepara1on for Future Learning (PFL) [Bransford & Schwartz, 1999]

Good Experience

Future Learning

Bad (or no) Experience

Sequestered-‐Problem-‐Solving test

Prepara1on-‐for-‐Future-‐Learning test


That’s where the “Prepara1on for Future Learning” part of my 1tle comes in. PFL is an assessment framework designed to measure inchoate forms of prior knowledge that tradi1onal (or “sequestered-‐problem-‐solving”) tests miss.

This slide shows a generalized diagram of the forest-‐test-‐lecture-‐test scenario I just described: some learners have a good (which is to say, learning-‐relevant) experience, and others don’t. On a Sequestered-‐Problem-‐Solving test, they look about the same. But if you then provide a learning opportunity that is designed to mold that prior experience into a formal knowledge structure and test ‘em again, you can detect the benefits (or lack thereof) of the experience.

Study Design


In a moment I’ll talk about how I used the PFL framework, but first, a point about my study-design goals. Reports on the state of the field, like the 2011 National Research Council report, describe the evidence for games supporting schoolish learning as “emerging”, “inconclusive”, and “very limited”, with “gaps and weaknesses” that “make it difficult to...demonstrate their effectiveness...”These statements reflect the fact that many stakeholders want from the GLS community something like an FDA study: a randomized field trial with an intention-to-treat analysis and very traditional, schoolish operationalizations of learning. So I decided to try to run one.

Study Design


Here’s a first pass at the study design, stripped down so you can see the parallels with the forest-field trip example (more details will follow).

I randomly assigned community-college students to three conditions: play Civilization 4, play Call of Duty 2, or play no game. (I just gave gameplay participants the games they’d been assigned and told ‘em to play at home however they normally play, for at least 15 hours over the course of about 5 weeks.)

Study Design


Then I had all participants come in and take a 16-item multiple-choice test about World War II history.

Study Design


Then I had them watch a 20-minute narrated-slideshow lecture about World War II history.

Study Design


Then I gave ‘em another multiple-choice test about World War II history (this time 36 items).

Study Design


So that’s the basic study design. Here I’ll fill in a few more details.

Study Design


First, the participants were 102 local community-college students (Control: n = 33; CoD2: n = 34; Civ4: n = 35; 16-42 yrs, median 20 yrs; 64% female) whom I compensated with course credit and (if they played for the full 15 hours) pay; all had completed a huge demographic questionnaire (roughly 280 questions) as part of their research-participation program; and the way I explained the study was that everyone would get a free game and (possibly) a gift card, with the only differences being *which* game and *when* the gameplay would occur (before or after the in-person session); this way Control participants wouldn’t feel shortchanged.

Study Design


The way I verified gameplay was by collecting and analyzing participants’ save-game files (auto-generated by games so players can pick up where they left off).

Study Design


All players whose save-game files showed evidence of at least 15 hours of gameplay were compensated with $75 gift cards (45 people earned ‘em: 11 Control, 15 CoD2, 19 Civ).

(And you can see here that the “control” participants did receive a game to play for 15+ hours and got compensated if they did so.)

Materials


Now I’ll describe the games, lecture, and measures.

The Games


I chose these games because they were (a) both really popular with players and critics when they were released in 2005; (b) from successful franchises of games; (c) and old enough to be playable on any modern computer but still new enough to seem “cool”.

The Games: Civilization IV


Civ4 is a turn-based-strategy game.



You play as the immortal, autocratic ruler of a civilization, and your task is to guide your people through roughly 6000 years of history by making lots of choices.



You’re plopped down on some arbitrary Earth-like world, and you settle cities, build infrastructure, engage with other civilizations through diplomacy and/or warfare, and create wonders based on those in Earth’s history (like the Parthenon, or Rock ‘n’ Roll, or the Manhattan Project).



By the end of the game, you’ll have built a bunch of cities, fought some wars, and made a ton of choices.

The Games: Call of Duty 2


CoD2 is a first-person-shooter game.



You play as a lowly soldier: a Soviet peasant repelling the German invasion; then later a Brit in the North African campaign; and finally an American in the invasion of France, ending the game by crossing the Rhine into Germany.



In contrast to Civ4, CoD2 is a real-time game in which you are required to navigate a 3D environment and shoot things (like teddy bears).



You play as part of a small squad of soldiers overcoming various obstacles to reach the next objective, which is marked as a gold star on your map (lower left). CoD2 gameplay is fast, twitchy, and visceral, not much time for thoughtful reflection—you just shoot whatever threatens you and move toward the next gold star. But it’s all happening in the historical context of WWII theaters of war.

The Lecture


The lecture covered WWII from the initial troubles in Asia in the 1930s to the dropping of the atomic bombs. My primary resource for the lecture was a SparkNotes guide (like Cliff’s Notes: high-schoolers might use it to study for their history tests). I wrote the lecture to cover all of WWII but also to focus on two sets of themes, corresponding to the gameplay experiences that I predicted the two games would produce.

The Lecture


I had hoped that (because of their gameplay experiences) Civ4 players would engage more with the Nations themes and CoD2 players would engage more with the Battles themes.

The Tests[SparkNotes] Pre2.1 From the perspective of Western leaders, Stalin’s actions as leader of the Union of Soviet Socialist Republics reflected an emphasis on which of the following concepts?individualismfreedomhuman dignityaggression

[NAEP] Post2.31 When the United States entered the Second World War, one of its allies wasGermanyJapanthe Soviet UnionItaly

[CST] Post2.36 The purpose of the Manhattan Project was toprovide economic aid to Latin American countriesdevelop atomic weapons for the U.S. militarybring about an end to poverty in U.S. urban areasoffer assistance to relocated European refugees


I built the pre- and post-lecture tests using traditional multiple-choice items that I pulled from three sets of standardized tests: a quiz from the back of the SparkNotes guide I mentioned; the National Assessment of Educational Progress; and the California Standards Tests. Here are three items.

Open-Ended QuestionsAfter post-lecture test, two scenarios not mentioned in the lecture


So these traditional tests were my nod to the conservative folks out there who think that learning is factual retrieval. But we at GLS know better! So in addition to these traditional tests, I also included two sets of open-ended questions that described scenarios not mentioned in the lecture.


One for Nations, one for Battles


One of these two scenarios was designed to pick up on a focus on the Nations themes I had tried to build into the lecture (and hence to favor Civ4 players), while the other was designed to pick up on a Battles focus (and hence favor the CoD2 players).


One for Nations, one for Battles

Two questions per scenario:

What’s going on?

What would you want to ask to learn more?


Participants were asked what they thought was going on in each scenario and, more importantly, what questions they’d want to ask to learn more.

Open-Ended Questions(Nations focus)

In 1940, in Mers-el-Kebir, Algeria, commanders of some British ships spoke with commanders of some French ships, and then the British ships fired on the French ships, sinking the ships and killing over 1,200 French sailors.

Why do you think this might have happened? (Feel free to guess.)

What questions would you ask to figure out why this happened? (Don't just say, "I would ask why this happened." That's too easy. Think about what kinds of facts about the situation you would want to know.)


Here’s the first scenario (the Nations-focus one) and its two questions…

Open-Ended Questions(Battles focus)

On June 6, 1944, an American Ranger battalion landed on the beach at the foot of the cliffs of Pointe du Hoc, in France. They then climbed those cliffs under fire from the Germans to destroy a set of large artillery guns.

Why do you think the Americans wanted to destroy the guns? (Don't just say, "To stop the Germans from using them." Be specific. Think about where the Germans might have wanted to use the guns.)

What questions would you ask to figure out why this happened? (Don't just say, "I would ask why this happened." That's too easy. Think about what kinds of facts about the situation you would want to know.)


…and here’s the Battles-focus scenario and questions.

Results


My analysis protocol for the multiple-choice tests was to fit ANCOVA models using test scores as the outcome variables. To choose predictors, I created a candidate list of nine that I had reason to believe might be useful...

Candidate predictors for ANCOVA models

• Gameplay condition


Gameplay condition (operationalized differently by analysis)



• Quarter of data collection


quarter of data collection (to account for cohort effects)




• Gender


gender




• Gender

• Age


age




• Gender

• Age

• Prior gameplay history


prior-gameplay history (4-level ordinal from “never” to “> 6 times”)




• Gender

• Age


• English proficiency level


English proficiency (lots of non-native speakers in my sample)




• Gender

• Age



• Prior social-studies interest


prior social-studies interest (5-level Likert)




• Gender

• Age




• Enjoyment of the assigned game


enjoyment of the assigned game (obviously only relevant for gameplay participants)




• Gender

• Age




• Enjoyment of the assigned game

• Pre-lecture-test scores (for post-lecture-test ANCOVA)


and pre-lecture-test scores (obviously only for the post-lecture-test ANCOVA).

I then fed this candidate set of predictors into an all-possible-subsets selection procedure, which examines all combinations of the predictor set to find the model with the highest adj-R^2. To avoid capitalizing on spurious patterns in the data set, I also constrained it to include only models all of whose predictors were at least marginally significant. I call the resulting model the “parsimonious” model.

Parsimonious model for pre-lecture-test scores

Source df SSTypeIII F η2 p

English proficiency 1 70.86 13.32 .12 .00042**

R2adj = .11, F(1, 100) = 13.32, p = .00042

Residuals 100 532.16


The only predictor of pre-lecture-test scores was English proficiency. This is the SPS test from our PFL model. NOTE: If this were all we did to test the learning benefits of recreational commercial gameplay, we’d find no benefit. But on the post-lecture-test...

Parsimonious model for post-lecture-test scores

Source df SSTypeIII F η2 p

Received a game 1 88.97 4.05 .026 .047*Quarter 2 125.40 2.85 .036 .063.Age 1 123.55 5.63 .036 .020*English proficiency 1 104.52 4.76 .030 .032*Prior SS interest 4 334.12 3.80 .096 .0068**Game enjoyment 4 239.07 2.72 .069 .035*Pre-lecture test 1 145.08 6.60 .042 .012*

R2adj = .36, F(14, 87) = 5.06, p < .0001

Residuals 87 1910.95


...all of a sudden a lot is going on. The key points for this talk are (a) many things are involved in how players will learn from gaming experiences, and (b) gameplay participants significantly outscored control participants—i.e., they learned more from the lecture.

In fact, gameplay participants scored about 6% higher on the post-lecture test than did control participants (without considering covariates, just a straight means comparison). In the language of school, that translates to over four percentage points on an exam (74.3% for control participants and 78.6% for gameplay participants), or nearly half of a letter grade. Cohen’s d = .27, which is substantial for a randomized field trial.

No 22 16 26 Yes 11 18 9

Battles focus Control CoD2 Civ4

Fisher’s exact test: p = .030

No 22 21 15 Yes 11 13 20

Nations focus Control CoD2 Civ4

Fisher’s exact test: p = .058

Responses to open-ended questions


The other outcome measure, remember, was participants’ responses to open-ended questions about novel WWII scenarios. It turned out that participants’ gameplay experiences affected the focus of their responses, with Civ4 participants adopting a more global “Nations” focus and CoD2 participants adopting a more local “Battles” focus.

(The specific operationalization of my coding scheme for this scenario was to code participants as having a “Nations” focus if and only if (a) the participant’s questions mentioned Resources (including territory), Empires (including colonies), Defenses (including enemies), or Alliances (including treaties), or (b) the participant wrote of the actors as being the nations themselves (e.g., Britain, France) rather than agents of those nationalities (e.g., British commanders, French ships).

The operationalization of my coding scheme for this scenario was to code participants as having a “Battles” focus if and only if the participant’s questions mention (a) Weaponry (including capabilities of particular weapons), Terrain (including avenues of ingress for the engagement), Communication (but not including prior intelligence about the engagement), or Objectives (but not including consequences of the engagement) or (b) such tactical elements as the time course of the engagement, casualties, or troop size. (CoD2 participants’ responses tended to reflect the in-the-moment viewpoint of a soldier anticipating climbing those cliffs to engage an enemy.))

Discussion


I’ve got three basic takeaways from these results, and then three suggestions for various stakeholders.

Summary of Findings

• Playing enjoyable video games at home can help students learn in school(ish

settings)


First, the results of this study support the claim that playing enjoyable video games at home can help both male and female students learn in school, if the formal instruction leverages the students’ gameplay experiences. (The strong predictive effect of prior social-studies interest shows the importance of also leveraging students’ interests.)

Summary of Findings


settings)

• Different game experiences lend themselves to different types of instruction


Second, the results from the open-ended questions underscore the notion that different games will offer different types of experiences that prepare players preferentially for different topics of formal instruction.

Summary of Findings


settings)

• Different game experiences lend themselves to different types of instruction

• Gameplay can influence both retention of facts and choices about what to learn


And third, the open-ended-question results further suggest that these gameplay experiences can improve not only retention of facts presented by direct instruction but also students’ choices about what to learn.

Considerations for Practice

• Curriculum Designers: From task analysis of games to curriculum


Now, my three recommendations for folks who want to cash these results out in some way.

First, for folks who want to build curricula to leverage gameplay experiences, I’d suggest a careful task analysis of gameplay to determine the relevant properties of its experiences (e.g., for Civ4 it was thinking as a nation; for CoD2 it was probably as simple as just being exposed to the historical context of WWII).



• Educators: Gameplay doesn’t have to be wasted time


For educators who have to deal with their students playing games for hours each week, I’d suggest that they recognize that gameplay is pervasive and powerful and that they embrace it (by tying the compelling experiences found in games with the powerful explanatory structures found in the standard curriculum).



• Educators: Gameplay doesn’t have to be wasted time

• Game Designers: You can be chickens, not pigs


And for commercial game designers—who know how hard it is to make a good game, let alone a good learning game, and have therefore steered clear of the educational game space—I’d say that this study suggests that they needn’t try to cram all of the curricular content into the game itself. Instead, they can continue to let the game do what it does best (provide great experiences) with perhaps some small tweaks here and there to better serve as foundations upon which educators might build.

(from a joke about eggs/bacon for breakfast: the chicken is interested, but the pig is committed)

Acknowledgement

Financial support for this dissertation was provided by a

SUSE Dissertation Support Grant and a Gerald J.

Lieberman Fellowship



And that’s it—thanks very much for your time!

Questions?

Commercial Video Games as Preparation for Future Learning

Education

Transcript of Commercial Video Games as Preparation for Future Learning