Using Hierarchical Reinforcement Learning to Solve a Problem with Multiple Conflicting Sub-problems

Reinforcement Learning

• Involves an agent interacting with an environment

• The agent can be in one of various states in the environment

• The agent is not told which action is correct, but is given a measure of an action for a given state

• After a while the agent develops a policy

The curse

• As complexity of the environment grows, state space increases exponentially

• We can try to cleverly reduce state space

• Hierarchical reinforcement learning

Hierarchical Reinforcement Learning

• A complex problem can often be broken up into multiple conflicting sub-problems

• Hierarchical reinforcement learning can handle this

• Deals with each sub-problem separately using reinforcement learning

• Decides which sub-problem to attempt next using reinforcement learning

A Practical Example: The Mars Lander

Perform Various Conflicting Tasks:

• Explore the terrain• Collect soil

samples• Return to base for

refuelling

My Project

• Apply hierarchical reinforcement learning to a complex problem

• Consist of an agent existing in an environment where it will have to achieve an overall goal

• Agent will be a primitive creature trying to survive in the wilderness

My Project

• The overall goal will be for the creature to remain happy or comfortable in the wilderness

• Overall goal can be divided into sub-goals• These sub-goals will be:

– Eating food– Drinking water– Resting under a Shelter– Repairing Shelter– Avoiding hazards

The Gridworld

Motivation for this approach• X pos Y pos Hunger Thirst Fatigue Shelter Condition

• 13 x 13 x 10 x 10 x 10 x 10

= 1690000 Possible states• Sub-goals separated out:• (Xpos, Ypos, hunger) , (Xpos, Ypos, Thirst)

(Xpos, Ypos, Fatigue), (Xpos, Ypos, Shelter Condition)

• (13 x 13 x 10) x 4

=1690 x 4 = 6760 Possible states