Motivated Reinforcement Learning for Non-Player Characters in Persistent Computer Game Worlds

1Intelligent Database Systems Lab

國立雲林科技大學National Yunlin University of Science and Technology

Motivated Reinforcement Learning for Non-Player Characters in Persistent

Computer Game Worlds

Advisor : Dr. HsuPresenter : Chia-Hao YangAuthor : Kathryn Merrick, Mary Lou Maher

SIGCHI 06


N.Y.U.S.T.

I. M.

Motivation Objective Introduction Method Experiments Discussion Conclusions Habituation SOM Q-learning

Outline


N.Y.U.S.T.

I. M.

Motivation Many NPC possess a fixed set of pre-programmed

behaviors and lack the ability to adapt and evolve in time with their surroundings.


N.Y.U.S.T.

I. M.

Objective To create NPC that can both evolve and adapt with

their environmental.


N.Y.U.S.T.

I. M.Introduction Current technologies for NPCs

─ Reflexive agents Only recognized states will produce a response

State machines & rule-based approaches EX : Baldur Gate & Dungeon Siege

─ Learning agents It can modify their internal structure to respect to some task.

Black and White─ Reinforcement learning agents

The agent records the reward signal. Then chooses an action which attempts to maximize the long-r

un sum of the values of the reward signal. Tao Feng


N.Y.U.S.T.

I. M.Method

Motivated reinforcement learning agents─ It use a motivation function to directs learning.─ Skill development is dependent on the agent’s environment & these

skills are developed progressively over time.

S(t) – S(t-1)

S(t-1) – S(t-2)

Q-learning


N.Y.U.S.T.

I. M.Experiments In order to experiment with MRL agent, we implemented a village s

cenario in Second Life.─ Support character

Trades people Location, object, inventory sensor Move to object, pick up object, use object effector Ex : the pick, when used on the mine, will produce iron which can converted to weapon

s when used near the forge


N.Y.U.S.T.

I. M.Experiments

─ Partner character Vendor character

Location, object sensor Move to object effector Ex : In Ultima Online players can set up vendor characters to sell the goods they have c

rafted.


N.Y.U.S.T.

I. M.Conclusions This paper has presented MRL agents as a means of

creating non-player characters which can both evolve and adapt.

MRL agents explore their environment and learn new behaviors in response to interesting experiences, allowing them to display progressively evolving behavioral patterns.


N.Y.U.S.T.

I. M.Habituation SOM An HSOM consists of a standard Self-Organizing Map with an

additional habituating neuron connected to every clustering neuron of the SOM.

─

─


N.Y.U.S.T.

I. M.Q-Learning

It’s a part of reinforcement learning algorithm which has been widely used for many applications such as robotics, multi agent system, game, and etc.

It allows an agent to learn through training without teacher in unknown environment.─ Modeling the Environment

─ putting similar matrix name Q in the brain of our agent

reference


N.Y.U.S.T.

I. M.Q-Learning ─ algorithm

─ example

reference

……

Motivated Reinforcement Learning for Non-Player Characters in Persistent Computer Game Worlds

Documents

Transcript of Motivated Reinforcement Learning for Non-Player Characters in Persistent Computer Game Worlds