Exploration Strategies for Learned Probabilities in Smart Terrain Dr. John R. Sullins Youngstown...

Exploration Strategies for Learned Probabilities in

Smart Terrain

Dr. John R. Sullins

Youngstown State University

John Sullins Youngstown State University

Exploration Strategies for Learned Probabilities in Smart Terrain

2

Problem Definition

• Agent given a map of world

• Map gives locations where goals may possibly be

• Different categories of locations have different probabilities



3

Learned Probabilities

Problem:• Agent does not know these probabilities• Agent must learn them from examples [a, b] of

that category

ai = number of past examples of category Ci where

goal has been present

bi = number of past examples of category Ci where

goal has not been present



4

Learning with Costs

• Agent must physically move to a target to know whether it meets goal

• Cost usually proportional to distance traveled



5

Learning with Costs

Knowledge gained by exploring target

Cost of exploring targettradeoff

Requires a rational strategy for exploration



6

Outline

• Learning as reducing future costs

• Beta functions and probabilistic smart terrain

• Defining an information gain function – Estimating extra distances traveled due to errors– Factoring in category prevalence

• Creating an influence map for agent movement

• Benchmark and empirical testing



7

Exploration Strategy

• Main idea: Exploration now reduces travel time in future

– t1 is instance of category C1 with prior knowledge [a1, b1]

– t2 is instance of category C2 with prior knowledge [a2, b2]

Agentt1 t2d d



8

Value of Information

• Rational action: Move to target in more probable category first

• Problem:Agent must estimate probabilities from examples

• Fewer examples Greater likelihood estimate wrong

Agentt1 t2d d



9


• Probabilities estimated from limited data: p1

estimate = 0.15 p2estimate = 0.2

– Agent will move towards t2

• Suppose actual probabilities different: p1

actual = 0.25 p2actual = 0.1

• Would have been better to move to t1 first

Agentt1t2



10


• Agent will have to backtrack to t1 if goal not met by t2

• Expected distance traveled will be greater than if moved towards t1 first

• Better estimates of probabilities less travel time

Agentt1

t2



11

Outline








12

Beta Distribution

• Estimate of probability category meets goal given examples [a, b] of category

beta[a, b]() = α a -1 b -1

• “Liklihood” the actual probability is given [a, b]

• Best estimate of actual probability = Exp(beta[a, b]() )



13

Beta Distribution

• “Narrows” as more examples explored

• More examples less error in estimate of



14

Probabilistic Smart Terrain

• Agent movement in worlds where targets have probability of meeting goal– pi : probability target i meets goal

– di : distance (in moves) from agent to target i

– Based on targets within dmax moves

• For each adjacent tile, computes expected distance to some target that meets goal



15

Probabilistic Smart Terrain

• Expected number of moves character must travel from x to target that meets goal

dmax

Dist(x) = Σ (1 – pi ) d di < d

Probability no target within d moves of x meets goal

(assumption of conditional independence)

Summed over all distancesup to some maximum dmax

(otherwise sum could be infinite)



16

Probabilistic Smart Terrain• Compute expected distance Dist(x) for all tiles x• Agent moves to adjacent tile with lowest Dist(x)



17

Outline








18

Simple Two-target Case

• Simple case where agent must “choose” between two targets to explore

– ti is instance of category Ci with prior knowledge [ai, bi]

– tJ is instance of category Cj with prior knowledge [aJ, bJ]

• Targets equidistant at distance d

• d is average distance between targets in world

Agentti tJdd



19

Estimating Distance Traveled

• Assume ti has higher estimated probability(Exp(beta[ai, bi](i) ) > Exp(beta[aj, bj](j) )

• Expected distance traveled: Dist(i, J) = d + 2d(1 - i) + (dmax - 3d) (1 - i) (1- J)

Agentti tJdd

Move to ti Backtrack to tj if ti does not meet goal

Case where neither ti nor tj meet goal



20

Defining an Error Function

i, J may take on many values– Likelihood of a

particular defined by beta()[a, b]

• Moving to ti first is error in cases where i < J

Ci

CJ

i J



21


• Amount of error for given (i, J) defined as

ErrDist(i, J) = Dist(i, J) - Dist(J, i)

= 2d(J - i) if J > i

0 otherwise

Expected distance if move to ti first

Expected distance if move to tj first



22


• Error weighted by likelihood of i, J (as defined by beta function)

ErrPair([ai, bi], [aJ, bJ]) =

0 0 ErrDist(i, J) beta[ai, bi](i) beta[aJ, bJ](J) i J

Total error possible given these examples of Ci and Cj

Summed over all possible combinations

of i, J weighted by their likelihoods



23


• Additional values of [a, b] narrow the beta distributions

• Narrow distributions allow less error

• P(i < J ) much

smaller

CJ

Ci



24


• Categories with similar [a, b] may still overlap

• However, i and j

will likely be similar even if i < j

• ErrDist(i, j) will be very small

Cj

Ci

i j



25

Outline








26

Category Prevalence

• Prioritize instances of more prevalent categories

– ti category Ci with |Ci| instances in world

– tJ category CJ with |CJ| instances in world

– |Ci| >> |CJ| (many more instances of Ci)

• More benefit to be gained by exploring ti

Agentti tJ



27

Category Pair Likilihood

• Agent is between two targets in different categories

• What is likelihood those categories are Ci and Cj?

• L(Ci, Cj) = |Ci| |CJ| + |Ci| |CJ| |Ctotal| (|Ctotal| - |Cj|) |Ctotal| (|Ctotal| - |CJ|)

• Ctotal = total number of targets in all categories

Agentti tJdd



28

Category Error Measure

• Total error measure for category Ci based on relationship to all other categories CJ :

– Error ErrPair([ai, bi], [aJ, bJ]) relative to that category

(based on overlap of their beta functions)

– Likelihood L(Ci, CJ) agent must choose between two

targets in those categories

ErrCat(Ci, [ai, bi]) = ErrPair([ai, bi], [aJ, bJ]) L(Ci, CJ) i ≠ J



29

Defining Information Gain

• Information gain from exploring instance of Ci

How incrementing [ai, bi] would decrease ErrCat(Ci, [ai, bi]) by narrowing the beta function

• Gain(Ci, [ai, bi]) ) = ErrCat(Ci, [ai, bi]) – ErrCat(Ci, [ai′, bi′])

Current error before target explored

Estimated error if target were explored



30

Defining Information Gain

• Problem: Do not know whether given target meets goal until explored

– Do not know whether it would increment ai or bi

• Solution: Estimate from current expected value Exp(beta[ai, bi](i))

[ai′, bi′] = [ai + Exp(beta[ai, bi](i)),

bi + (1 - Exp(beta[ai, bi](i)))]



31

Example of Information Gain

• Example: Information gain for [2, 6] and [4, 4]– Same prevalence, average distance = 10

New Examples

Category [4, 4]

Category [2, 6]

1 1.941 2.1192 1.597 1.7273 1.336 1.4354 1.133 1.2125 0.972 1.038



32

Prior Category Knowledge

• More existing examples Less valuable future examples become

• Preference given to categories about which less is known



33

Outline

• Learning as reducing future travel costs







34

Influence Maps

• Targets influence nearby agents– Influence = information gain

of target category

• Influence decreases with distance from target

• Agent moves in direction of increasing influence



35

Falloff Function

• Inverse function used to decrease influence over distance

Influence(t) = Gain(Ci, [ai, bi])) 1 + t / d

t = distance in tiles

d = average distance between targets



36

Combining Influences

• Question: How should influences from multiple targets be combined

• Goal: Prioritize exploring groups of targets

– |Ci| ≈ |Cj| ≈ |Ck|

– |[ai, bi]| ≈ |[aj, bj]| ≈ |[ak, bk]|

• Can quickly explore both ti and tk by moving left

Agentti tjtk

Prior information and prevalence similar



37

Additive Combined Influences

• Influences from targets in different categories added to compute total influence at a tile

• Inverse falloff function chosen to minimize possibility of local maxima in influence map



38

Influences in Single Category

• Information gain decreases for each target explored in same category

• Decrease must be factored into influence map

Agent ti1

ti3

ti2

2.1192.119

2.119

1.727

1.435



39

Computing Total Influence

• Influence at tile t from all targets in category Ci:

TotalInfluence(t, i) = Gain(Ci, [ai, bi], k) k 1 + tk / d– tk = distance to kth

nearest target

– Gain(Ci, [ai, bi], k) = expected information gain from kth example

• Influence at tile t from targets in all categories:

TotalInfluence(t) = Gain(Ci, [ai, bi], k) i k 1 + tk / d



40

Updating the Influence Map

• Influence map computed for all tiles in area of agent

• Agent moves in direction of increasing influence until some target ti reached

• Agent determines whether target meets goal, and either increments ai or bi for category Ci

• Information gain recomputed for all categories

• Influence map recomputed (with ti removed)



41

Updating the Influence Map



42

Outline

• Learning as reducing future travel costs







43

Prior Knowledge Benchmark

• Instance of category with knowledge [1, 2]• Instance of category with knowledge [2, 4]

– Category prevalence similar

• Agent should move towards instance of category with less knowledge



44

Category Prevalence Benchmark

• Instance of category with two instances• Instance of category with single instance

– Prior knowledge of both = [1, 2]

• Agent should move towards instance of category with greater prevalence



45

Much Closer Distance Benchmark

• Knowledge = [10, 15] and prevalence = 7• Knowledge = [8, 12] and prevalence = 8

• Even though further target has better information gain and prevalence, agent should move towards significantly closer targets



46

Large-scale Testing

• 30 x 20 world (with obstacles)

• 4 categories of targets

• Targets placed randomly for each trial

• Probability tile contains target = 0.05



47

Category Data

Preva-lence

Actual probability

Prior knowledge

[a, b]

A 0.2 0.1 [10, 90]

B 0.2 0.1 [1, 3]

C 0.3 0.25 [1, 5]

D 0.3 0.25 [25, 75]

High priority

due to information gain

Somewhat high priority due to category prevalence



48

Importance of Learning

• Limited category data can cause errors in estimated probabilities

• This can lead to incorrect decisions about which target to move to next

Actual P Prior knowledge

A 0.1 [10, 90]B 0.1 [1, 3]C 0.25 [1, 5]D 0.25 [25, 75]

Overestimates probability of B –moves towards instances too often

Underestimates probability of C – ignores instances too often



49

Does the Learning Strategy Work?

• 100 trials with targets randomly placed• For each trial, agent given 50 moves for learning

– Influence map generated– Agent followed influence map to target– Actual probabilities used to update [a, b] for that

category– Information gains updated and map recomputed

• Question: Which categories were explored most?



50

Does the Learning Strategy Work?

• Average number of each category explored per trial:

Category Average explored per trial

A 1.17

B 2.52

C 3.66

D 2.10

Greater information gain

Higher prevalence



51

Is the Learning Strategy Useful?

• Does the information gain strategy reduce future search time for targets that meet goals?

• Comparison of results to simpler “naïve” strategy– During learning phase, simply

move to closest target instead of computing information gains



52

Training and Testing

• Training phase:– Learning strategy (information gain or naïve) used to

move agent 50 moves

– Each time target in category Ci reached, update its [ai, bi] based on actual category probabilities

– Product of learning: estimated probabilities pi for each

category computed as Exp(beta[ai, bi](i)



53

Training and Testing

• Testing Phase:– Agent placed at every location in world

(536 non-wall tiles)

– Existing probabilistic smart terrain algorithm used to search for a target that meets goal from that point

• Based on estimated probabilities from training phase

• Question: How many moves were required on average to find a goal?



54

Results of Testing

• 100 trials using both naïve and information gain learning

• Information gain learning focused on categories about which less was known (B and C)

• More accurate estimated probabilities• Less travel time due to moving to wrong targets

Strategy Average tiles explored until goal found

Information gain 5.294Naive 6.473



55

Ongoing Work

• Learning while acting to meet goals– Agent must meet current needs

(which presumably have some urgency)– Agent must also explore to learn knowledge to better

meet future needs

TradeoffCosts of not meeting current needs while exploring

Costs of extra travel in future if exploration not done now



56

Ongoing Work

• Learning in hierarchical worlds– Agent does not know exact location of all targets– Agent only knows expected number in a given region– Will not know what region actually contains until move

to it

AExp(C1) = 3.2

Exp(C2) = 2.4

Exp(C1) = 1.7

Exp(C2) = 4.5

? ?

Exploration Strategies for Learned Probabilities in

Smart Terrain

Dr. John R. Sullins

Youngstown State University

Exploration Strategies for Learned Probabilities in Smart Terrain Dr. John R. Sullins Youngstown...

Documents

Transcript of Exploration Strategies for Learned Probabilities in Smart Terrain Dr. John R. Sullins Youngstown...