Research Related to Real-Time Strategy Games

Robert Holte

November 8, 2002

No Games Knowledge, but…

• Single-agent search = sequential decisions with additive costs– path-finding occurs in RTS games– are there other occurrences ?– transfer from 1-agent to 2-agent or N-agent games ?– is a 2-agent game with a deterministic opponent a

single-agent search problem ?

• Path-finding research group (P. Yap, Yngvi, …)

and …

• Combinatorial optimization = resource allocation with complex constraints– combinatorial auctions– “boolean programming”– hill-climbing, ESG, a bit of CPLEX

and …

• Machine Learning– opponent-modelling (poker)– user-modelling (adaptive games, personalization)– learning to make good moves (M. Mueller)

• Interactive Systems– mostly-automatic evaluation methodology

• Abstraction in Poker

Using Abstraction to Speed Up Search

Search = path-finding in a graph

GIVEN:– A graph– Two nodes in the graph (“start”, “goal”)

FIND:– a shortest path from start to goal

Labyrinth at the gardens of Versailles(could be a computer game!)

“Implicit” Graphs

Instead of an adjacency list/matrix,Define a graph by a successor function, succ,

and one or more “seed” nodes.Nodes of the graph = transitive closure of succ applied to the seed nodes

An edge exists from n1 to n2 iff n2 succ(n1)

A graph can be exponentially larger than its description.

Towers of Hanoi puzzlefor D disks:

– graph has 3D nodes– description is O(D2)

3-disk State Space

Speed up Search by Using Extra Information

• Solution “skeleton” = chain of subgraphs leading from start to goal– Refinement: search in the first subgraph until you reach

the second, then search in the second until you reach the third, etc.

– very fast but path found not guaranteed to be optimal

• Heuristic = h(n) = estimate of the distance from node n to the goal node.– if it never overestimates, it can eliminate (prune) parts of

the graph and still guarantee finding an optimal path

My Research

Aim:to generate heuristics or skeletal solutions automatically

Method:1. Create an abstraction of the given graph (simplified

version of your problem that “approximately” preserves “important” features of the problem).

2. Find a solution path in the abstract graph.3. Use the abstract solution as a skeletal solution, or use

its length as a heuristic.

Example: Towers of Hanoistart

goal

Abstraction:“Ignore the smallest

disk”start

goal

Abstract Solution

start

goal

Key Techniques

For Automatically Generating Abstractions– explicit graphs: STAR abstraction

– implicit graphs: domain abstraction

– select among abstractions using Korf & Reid’s method for predicting search time

Search Methods– refinement: AltO

– heuristic: Hierarchical A*, pattern databases

– anytime combination of refinement and heuristic search

STAR abstraction

Abstract state = connected subgraph

(all nodes within distance R of a given node)

• Fully automatic (but need to amortize cost)

• Fine control over granularity (R)

• Can be applied recursively (abstraction hierarchy)

• Guarantees refinement will succeed• Produces a consistent heuristic

AltO Refinement Technique

Conduct abstract search in the opposite direction to the base-level search;

Use the whole abstract search graph for refinement.

• 12% faster than standard refinement and produces better solutions (20% shorter)

• Robust – not sensitive to the abstract solution found, and not very sensitive to the radius of abstraction

• 10-40 times faster than A* (same abstraction) and solutions usually within 30% of optimal

Pattern Database Successes• Joe Culberson & Jonathan Schaeffer (1994).

– 15-puzzle– PDB + Manhattan Distance (MD) reduced MD-

alone search tree 1000-fold

• Rich Korf (1997)– PDBs enabled optimal solutions of Rubik’s Cube

• Stefan Edelkamp (2001)– PDBs competitive with best planners

• Istvan Hernadvolgyi– PDBs reduced macro-table solution lengths from 90

to 50

Example: 8-puzzle

1 2

3 4 5

6 7 8

Domain = blank 1 2 3 4 5 6 7 8

181,440 states

“Patterns”created by domain mapping

1 2

3 4 5

6 7 8

This mapping produces9 patterns

Domain = blank 1 2 3 4 5 6 7 8Abstract = blank

corresponding patternoriginal state

Pattern Database

Pattern

Distance to goal 0 1 1 2 2 2

Pattern

Distance to goal 3 3 4

Calculating h(s)

Given a state in the original problem

Compute the corresponding pattern

and look up the abstract distance-to-goal

8 1 4

3 5

6 7 2

2

Heuristics defined by PDBs are consistent, not just admissible.

Domain Abstraction

1 2

3 4 5

6 7 8 6 7 8

30,240 patternsDomain = blank 1 2 3 4 5 6 7 8Abstract = blank 6 7 8

Automatic Creation of Domain Abstractions

• Easy to enumerate all possible domain abstractions

• They form a lattice, e.g.

is “more abstract” than the domain abstraction above

Domain = blank 1 2 3 4 5 6 7 8Abstract = blank

Domain = blank 1 2 3 4 5 6 7 8Abstract = blank

8-puzzle: A* vs. PDB size#

node

s ex

pand

ed (

A*)

pattern database size (# of abstract states)

Hierarchical A*Unlike pattern databases, each h(n) is computed on

demand.

If h(n) is not already known compute it by searching at the abstract level.

The search at an abstract level can be informed by a heuristic defined by a higher abstract level.

* many abstract searches for one base-level search* naïve implementation is 10 times slower than

blind base-level search.

Speed versus Optimality

• Refinement– very fast (when refinability guaranteed)– suboptimal paths

• A*– much slower– optimal paths

• See also: bidirectional search, -admissible A*

Hybrid of HA* and Refinement (1)

Although the algorithms seem very different there are actually only two differences.

(1) The relative weight given to g and h.HA*: g + h

Refinement: h (use g to break ties)

Generalization: *g + (1- )*h ( 0 1 )

(2) What to do if a node n is encountered for which h(n) is not known from the first abstract search ?

HA*: compute h(n) by doing another abstract search

Refinement: ignore n

Generalization: choose randomly between the two strategies with probability P ( 0 P 1 )

Hybrid of HA* and Refinement (2)

2-d Space of Algorithms

0 0.5 1.0

= weight given to g

1.0

P

0.5

0.0

A*

Refinement

GraphTraverser

OptimalRefinement