Turn-Based Games

Héctor Muñoz-Avila

sources: •http://www.game-research.com/•www.gamespot.com•Wikipedia.org•Russell & Norvig AI Book; Chapter 5 (and slides)•Jonathan Schaeffer’s AAW 05 presentation •My own

Turn-Based Strategy GamesEarly strategy games was dominated by turn-based games

•Derivate from board games•Chess•The Battle for Normandy (1982)•Nato Division Commanders (1985)

Turn-based strategy:•game flow is partitioned in turns or rounds. • Turns separate analysis by the player from actions•“harvest, build, destroy” in turns• Two classes:

•Simultaneous•Mini-turns

Turn-Based Games Continues to be A Popular Game Genre

• At least 3 sub-styles are very popular:

– “Civilization”-style games

• Civilization IV came out last week

– Fantasy-style (RPG)

• Heroes of Might and Magic series

– Poker games

• Poker Academy

Some Historical Highlights

• 1952 Turing design a chess algorithm. Around the same time Claude Shannon also develop a chess program

• 1956 Maniac versus Human• 1970 Hamurabi. A game about building an economy for a

kingdom• The Battle for Normandy (1982)• 1987 Pirates!• 1990 Civilization• 1995 HoMM• 1996 Civilization II

• The best game ever?• …• 2005 Civilization IV• 2006 HoMM V

Side-tracking: Game Design: Contradicting Principles

• Principle: All actions can be done from a single screen.

• Classical example: Civ IV

• But: HoMM uses two interfaces: HoMM IV

Coming back: How to Construct Good AI?

• Idea: Lets just use A* and define a good heuristic for the game

Search space: a bipartite treeAfter all didn’t we use it with the 9-puzzle game?

• Problems with this idea:

Adversarial: we need to consider possible moves of our opponent (s)

Time limit: (think Chess)

Types of AdversarialTBGs (from AI perspective)

Perfect information

Imperfect information

Deterministic Chance

Chess, Go, rock-paper-scissors

Battleships, Stratego

Backgammon, monopoly

Civilization, HoMM

Bridge, Poker

Game tree (2-player, deterministic, turns)

Concepts:•State: node in search space•Operator: valid move•Terminal test: game over•Utility function: value for outcome of the game•MAX: 1st player, maximizing its own utility•MIN: 2nd player, minimizing Max’s utility

Minimax

• Finding perfect play for deterministic games

• Idea: choose move to position with highest minimax value

= best achievable payoff against best play

• E.g., 2-play game:

Minimax algorithm

Properties of minimax

• Complete?

• Optimal?

• Time complexity?

– b: branching factor

– m: # moves in a game

Yes (if tree is finite)

Yes (against an optimal opponent)

O(bm)

•For chess, b ≈ 35, m ≈100 for "reasonable" gamesTherefore, exact solution is infeasible

Minimax algorithm with Imperfect Decisions

evaluationFunction(state)

evaluationFunction(state)

Cutoff-test(state)

Cutoff-test(state)

Evaluation Function

• Evaluation Function

– Is an estimate of the actual utility

– Typically represented as a linear function:

EF(state) = w1f1(state) + w2f2(state) + … + wnfn(state)

– Example:Chess• weight: Piece Number

(w1) Pawn 1 (w2) Knight 3(w3) Bishop 3(w4) Rook 5 (w5) Queen 9

•Function; state Numberf1 = #(pawns,w) #(pawns,b)f2 = #(knight,w) #(knight,b)f3 = #(bishop,w) #(bishop,b)f4 = #(rook,w) #(rook,b)f5 = #(knight,w) #(knight,b)

Example: Evaluation Function

“all things been equal”White moves,Who is winning?

Is this consistent with Evaluation function?

Black

Yes!

Evaluation Function (2)

• Obviously, the quality of the AI player depends on the evaluation function

• Conditions for evaluation functions:

If n is a terminal node, Computing EF should not take longEF should reflect chances of winning

EF(n) = Utility(n)

If EF(state) > 3 then is almost-certain that blacks win

Cutting Off Search

• When to cutoff minimax expansion?

• Potential problem with cutting off search: Horizon problem

• Solution:

Fixed depth limitIterative deepening until times runs out

Decision made by opponent is damaging but cannot be “seen” because of cutoff

Quiescent: states that are unlikely to exhibit wild swings in the values of the evaluation functions

Example: Horizon Problem

“all things been equal”White moves,Who is winning?

Is this consistent with Evaluation function?

Black

No!

α-β pruning: Motivation

•A good program may search 1000 positions per second

•In a chess tournament, a player gets 150 seconds per move

•Therefore, the program can explore 150,000 positions per move

•With a branching factor of 34, this will mean a look ahead of 3 or 4 moves

• Facts:4-turns ≈ human novice

8-turns ≈ typical PC, human master12-turns ≈ Deep Blue, Kasparov

• How to look ahead more than 4 turns? Use α-β pruning

Example:

• Finding perfect play for deterministic games

• Idea: choose move to position with highest minimax value

= best achievable payoff against best play

• E.g., 2-play game:

α-β pruning

α-β pruning example

α-β pruning example

α-β pruning example

α-β pruning example

Principle of α-β Prunning

• α is the value of the best (i.e., highest-value) choice found so far at any choice point along the path for max If v α, max will avoid it

o Therefore, prune that branch

• β is the lowest-value found so far at any choice point along the path for min If v α, min will avoid it

o Therefore, prune that branch

The α-β algorithm

The α-β algorithm

Properties of α-β

• Pruning preserves completeness and optimality of original minimax algorithm

• Good move ordering improves effectiveness of pruning

• With "perfect ordering," time complexity = O(bm/2)Therefore, doubles depth of search

• Used in PC games today (9 moves look-ahead, Grand Master level)

Deterministic games in practice

• Checkers: Chinook ended 40-year-reign of human world champion Marion Tinsley in 1994. Used a precomputed endgame database defining perfect play for all positions involving 8 or fewer pieces on the board, a total of 444 billion positions.

• Chess: Deep Blue defeated human world champion Garry Kasparov in a six-game match in 1997. Deep Blue searches 200 million positions per second, 24 processors, quiescent identified with help of human grand masters

• Othello: human champions refuse to compete against computers, who are too good.

• Go: human champions refuse to compete against computers, who are too bad. In go, b > 300, so most programs use pattern knowledge bases to suggest plausible moves.

Additional Notes

• The next 5 slides are form David W. Aha (NRL) presentation at Lehigh University in Fall’04

Example Game: FreeCiv(Chance, adversarial, imperfect information game)

http://www.freeciv.org

Civilization II (MicroProse)

•Civilization II (1996-): 850K+ copies sold – PC Gamer: Game of the Year Award winner– Many other awards

•Civilization series (1991-): Introduced the civilization-based game genre

•Civilization II (1996-): 850K+ copies sold – PC Gamer: Game of the Year Award winner– Many other awards

•Civilization series (1991-): Introduced the civilization-based game genre

FreeCiv (Civ II clone)

•Open source freeware•Discrete strategy game•Goal: Defeat opponents, or build a spaceship

•Resource management – Economy, diplomacy,

science, cities, buildings, world wonders

– Units (e.g., for combat)•Up to 7 opponent civs•Partial observability

•Open source freeware•Discrete strategy game•Goal: Defeat opponents, or build a spaceship

•Resource management – Economy, diplomacy,

science, cities, buildings, world wonders

– Units (e.g., for combat)•Up to 7 opponent civs•Partial observability

FreeCiv ScenarioGeneral description

• Game initialization: Your only unit, a “settler”, is placed randomly on a random world (see Game Options below). Players cyclically alternate play

• Objective: Obtain highest score, conquer all opponents, or build first spaceship• Scoring: “Basic” goal is to obtain 1000 points. Game options affect the score.

– Citizens: 2 pts per happy citizen, 1 per content citizen– Advances: 20 pts per World Wonder, 5 per “futuristic” advance– Peace: 3 pts per turn of world peace (no wars or combat)– Pollution: -10pts per square currently polluted

• Top-level tasks (to achieve a high score): – Develop an economy– Increase population– Pursue research advances– Opponent interactions: Diplomacy and defense/combat

• Game initialization: Your only unit, a “settler”, is placed randomly on a random world (see Game Options below). Players cyclically alternate play

• Objective: Obtain highest score, conquer all opponents, or build first spaceship• Scoring: “Basic” goal is to obtain 1000 points. Game options affect the score.

– Citizens: 2 pts per happy citizen, 1 per content citizen– Advances: 20 pts per World Wonder, 5 per “futuristic” advance– Peace: 3 pts per turn of world peace (no wars or combat)– Pollution: -10pts per square currently polluted

• Top-level tasks (to achieve a high score): – Develop an economy– Increase population– Pursue research advances– Opponent interactions: Diplomacy and defense/combat

Game Option Y1 Y2 Y3

World size Small Normal Large

Difficulty level Warlord (2/6) Prince (3/6) King (4/6)

#Opponent civilizations 5 5 7

Level of barbarian activity Low Medium High

FreeCiv ConceptsConcepts in an Initial Knowledge Base

• Resources: Collection and useo Food, production, trade (money)

• Terrain: o Resources gained per turno Movement requirements

• Units:o Type (Military, trade, diplomatic, settlers, explorers)o Healtho Combat: Offense & defenseo Movement constraints (e.g., Land, sea, air)

• Government Types (e.g., anarchy, despotism, monarchy, democracy)• Research network: Identifies constraints on what can be studied at any time• Buildings (e.g., cost, capabilities)• Cities

o Population Growtho Happinesso Pollution

• Civilizations (e.g., military strength, aggressiveness, finances, cities, units)• Diplomatic states & negotiations

• Resources: Collection and useo Food, production, trade (money)

• Terrain: o Resources gained per turno Movement requirements

• Units:o Type (Military, trade, diplomatic, settlers, explorers)o Healtho Combat: Offense & defenseo Movement constraints (e.g., Land, sea, air)

• Government Types (e.g., anarchy, despotism, monarchy, democracy)• Research network: Identifies constraints on what can be studied at any time• Buildings (e.g., cost, capabilities)• Cities

o Population Growtho Happinesso Pollution

• Civilizations (e.g., military strength, aggressiveness, finances, cities, units)• Diplomatic states & negotiations

FreeCiv DecisionsCivilization decisions • Choice of government type (e.g., democracy)• Distribution of income devoted to research, entertainment, and wealth goals• Strategic decisions affecting other decisions (e.g., coordinated unit movement for trade)

• Choice of government type (e.g., democracy)• Distribution of income devoted to research, entertainment, and wealth goals• Strategic decisions affecting other decisions (e.g., coordinated unit movement for trade)

City decisions

Unit decisions

Diplomacy decisions

• Production choice (i.e., what to create, including city buildings and units)• Citizen roles (e.g., laborers, entertainers, or specialists), and laborer placement

– Note: Locations vary in their terrain, which generate different amounts of food, income, and production capability

• Production choice (i.e., what to create, including city buildings and units)• Citizen roles (e.g., laborers, entertainers, or specialists), and laborer placement

– Note: Locations vary in their terrain, which generate different amounts of food, income, and production capability

• Task (e.g., where to build a city, whether/where to engage in combat, espionage)• Movement

• Task (e.g., where to build a city, whether/where to engage in combat, espionage)• Movement

• Whether to sign a proffered peace treaty with another civilization• Whether to offer a gift

• Whether to sign a proffered peace treaty with another civilization• Whether to offer a gift

FreeCiv CP Decision SpaceVariables • Civilization-wide variables

o N: Number of civilizations encounteredo D: Number of diplomatic states (that you can have with an opponent)o G: Number of government types available to youo R: Number of research advances that can be pursued o I: Number of partitions of income into entertainment, money, & research

• U: #Unitso L: Number of locations a unit can move to in a turn

• C: #Citieso Z: Number of citizens per cityo S: Citizen status (i.e., laborer, entertainer, doctor)o B: Number of choices for city production

• Civilization-wide variableso N: Number of civilizations encounteredo D: Number of diplomatic states (that you can have with an opponent)o G: Number of government types available to youo R: Number of research advances that can be pursued o I: Number of partitions of income into entertainment, money, & research

• U: #Unitso L: Number of locations a unit can move to in a turn

• C: #Citieso Z: Number of citizens per cityo S: Citizen status (i.e., laborer, entertainer, doctor)o B: Number of choices for city production

Decision complexity per turn (for a typical game state)

• O(DNGRI*LU*(SZB)C) ; this ignores both other variables and domain knowledgeo This becomes large with the number of units and citieso Example: N=3; D=5; G=3; R=4; I=10; U=25; L=4; C=8; Z=10; S=3; B=10o Size of decision space (i.e., possible next states): 2.5*1065 (in one turn!)

o Comparison: Decision space of chess per turn is well below 140 (e.g., 20 at first move)

• O(DNGRI*LU*(SZB)C) ; this ignores both other variables and domain knowledgeo This becomes large with the number of units and citieso Example: N=3; D=5; G=3; R=4; I=10; U=25; L=4; C=8; Z=10; S=3; B=10o Size of decision space (i.e., possible next states): 2.5*1065 (in one turn!)

o Comparison: Decision space of chess per turn is well below 140 (e.g., 20 at first move)