Post on 18-Dec-2015
Using Hierarchical Reinforcement Learning to Solve a Problem with Multiple Conflicting Sub-problems
Reinforcement Learning
• Involves an agent interacting with an environment
• The agent can be in one of various states in the environment
• The agent is not told which action is correct, but is given a measure of an action for a given state
• After a while the agent develops a policy
The curse
• As complexity of the environment grows, state space increases exponentially
• We can try to cleverly reduce state space
• Hierarchical reinforcement learning
Hierarchical Reinforcement Learning
• A complex problem can often be broken up into multiple conflicting sub-problems
• Hierarchical reinforcement learning can handle this
• Deals with each sub-problem separately using reinforcement learning
• Decides which sub-problem to attempt next using reinforcement learning
A Practical Example: The Mars Lander
Perform Various Conflicting Tasks:
• Explore the terrain• Collect soil
samples• Return to base for
refuelling
My Project
• Apply hierarchical reinforcement learning to a complex problem
• Consist of an agent existing in an environment where it will have to achieve an overall goal
• Agent will be a primitive creature trying to survive in the wilderness
My Project
• The overall goal will be for the creature to remain happy or comfortable in the wilderness
• Overall goal can be divided into sub-goals• These sub-goals will be:
– Eating food– Drinking water– Resting under a Shelter– Repairing Shelter– Avoiding hazards
The Gridworld
Motivation for this approach• X pos Y pos Hunger Thirst Fatigue Shelter Condition
• 13 x 13 x 10 x 10 x 10 x 10
= 1690000 Possible states• Sub-goals separated out:• (Xpos, Ypos, hunger) , (Xpos, Ypos, Thirst)
(Xpos, Ypos, Fatigue), (Xpos, Ypos, Shelter Condition)
• (13 x 13 x 10) x 4
=1690 x 4 = 6760 Possible states