High-level robot behavior control using POMDPsJoelle Pineau and Sebastian Thrun

Carnegie Mellon University

CogRob2002 workshop Joelle Pineau

Introducing Pearl

• Pearl is a prototype nursing robot, providing assistance to both nurses and elderly people.

“thinkers”

eyes with cameras

sonar sensors

handlebars

wheeled base

carrying tray

LCD smile/frown

CogRob2002 workshop Joelle Pineau

Profile of our aging population

450,000 more nurses needed by 2008 campaign to recruit and retain nurses

and other health care providers

CogRob2002 workshop Joelle Pineau

Moving thingsaround

Moving thingsaround

Enabling use of remote

health services

Enabling use of remote

health services

Supportinginter-personal

communication

Supportinginter-personal

communication

Calling for helpin emergencies

Calling for helpin emergencies

Monitoring Rx adherence

& safety

Monitoring Rx adherence

& safety

Providinginformation

(TV, weather)

Providinginformation

(TV, weather)

Management support of

ADLs

Management support of

ADLsReminding

to eat, drink, & take meds

Reminding to eat, drink, & take meds

Providing physical

assistance

Providing physical

assistance

Linking the caregiver to resources

Linking the caregiver to resources

Our vision of robotic-assisted health-care

CogRob2002 workshop Joelle Pineau

The Nursebot project in its early days

CogRob2002 workshop Joelle Pineau

We need a high-level controller that can:

• prioritize goals from specialized modules

• trade-off goals with widely different costs/rewards

• trade-off between information-gathering and goal- satisfaction

• switch between tasks and share sensory information

• handle uncertainty

Speech recognition&synthesis(Ravishankar, 1996; Black et al., 1999)

Autonomous navigation(Burgard et al., 1999)

Autominder(Pollack et al., 2002)

People tracking/following(Montemerlo et al., 2002)

Top controller

CogRob2002 workshop Joelle Pineau

High-level robot behavior control usingPartially Observable Markov Decision Processes

USER + WORLD + ROBOT

ACTIONS

OBSERVATIONSBELIEF STATE

STATE

CogRob2002 workshop Joelle Pineau

What are POMDPs?

POMDP is n-tuple { S, A, , b, T, O, R }

POMDP task 1: State tracking:

After an action, what is the state of the world?

POMDP task 2: Computing a policy:

Which action should the controller apply next?

Very hard!

Not so hard.

Robot belief:

World state:

Control layer:

bt-1 ??

at-1

ot

st-1 st

...

...

??ot-1

...

...

rt-1 rt

CogRob2002 workshop Joelle Pineau

Our approach: Hierarchical POMDPs

• Key Idea: Exploit hierarchical structure in the problem domain to break a problem into many “related” POMDPs.

Act

InvestigateHealth

Move

NavigateCheckPulse

AskWhere

North South East West

CheckMeds

subtask

abstract action

primitive action

CogRob2002 workshop Joelle Pineau

Planning with Hierarchical POMDPs

• Given POMDP model M = { S, A, , b, T, O, R } and hierarchy H

• For each subtask h H:1) Set components

Ah children nodes

Sh S

h bh, Th, Oh, Rh

2) Minimize modelSh {zh(s0), …, zh(sn)}

h {yh(o0), …, yh(op)}

3) Solve subtask hh {bh, Th, Oh, Rh}

Move

Navigate AskWhere

South East WestNorth

ANav ={N,S,E,W}SNav ={X,Y}Nav ={o0,…,om}

AMove ={AskWhere,Navigate}SMove ={X’,Y’,Destination}Move ={o0,…,op}

CogRob2002 workshop Joelle Pineau

Execution with Hierarchical POMDPs

• Step 1 - Update belief:

• Step 2 - Traversing hierarchy top-down, for each subtask:1) Get local belief.

2) Consult local policy.

3) If a is leaf node, terminate.

Else, go to that subtask.

Act

InvestigateHealth Move

NavigateCheckPulse

AskWhere

North South East West

CheckMeds

baoP

sbassPasoP

sbSs

jjii

ij

t

t

,|

,|,| 1

CogRob2002 workshop Joelle Pineau

Experimental Setup

• Task: Robot provides reminders and guidance to elderly user.

• Action hierarchy:

CogRob2002 workshop Joelle Pineau

Sample Scenario

CogRob2002 workshop Joelle Pineau

Still unconvinced about the importance of uncertainty?

CogRob2002 workshop Joelle Pineau

On the question of numericanumerical vs logicallogical representations...

• We haven’t tried using logical representations to control the robot…

• But our experience tells us that:

Uncertainty is crucial when dealing with people.

Probabilistic techniques are necessary to reason about uncertainty.

Real belief tracking and planning really matters!