Towards robotic assistants in nursing homes: challenges and results

Towards robotic assistants in nursing homes:Towards robotic assistants in nursing homes:challenges and resultschallenges and results

Joelle PineauMichael MontemerloMartha Pollack *

Nicholas RoySebastian Thrun

Carnegie Mellon University*University of Michigan

The Nursebot Project Joelle Pineau

Introducing Pearl – A mobile robotic assistant for elderly people and nurses

cameras

sonars

handle bars

mobile base

carrying tray

LCD mouth

touchscreen

microphone& speakers

laser

ROLE:

Moving thingsaroundMoving thingsaround

Management support of ADLsManagement support of ADLs

Providing physical assistance

Providing physical assistance

Remotehealth servicesRemotehealth services

Supportingcommunication

Supportingcommunication

Calling for helpin emergenciesCalling for helpin emergencies

Monitoring Rxadherence & safety

Monitoring Rxadherence & safety

Providing info(TV, weather)Providing info(TV, weather)

Reminding to eat,drink, take meds

Reminding to eat,drink, take meds

Linking caregiverand resources

Linking caregiverand resources


The Nursebot project in its early days


Architecture

Cognitive supportNavigation Communication

High-level controller


• Localization and map building(Burgard et al., 1999)

• People detection and tracking(Montemerlo et al., 2002)

Architecture




• Autominder system (Pollack et al., 2002)

Architecture




• Speech recognition: Sphinx system(Ravishankar, 1996)

• Speech synthesis: Festival system(Black et al., 1999)

Architecture




The role of the top-level controller

• Établir les priorités parmi les objectifs des différents modules

• Négocier entre plusieurs objectifs ayant des coûts/gains variés

• Négocier entre l’acquisition d’information et la rencontre des objectifs

• Passer d’une tâche à l’autre en partageant l’information sensorielle

• Planifier malgré la présence d’incertitude


ACTION SELECTION - based on the trade-off between:

- goals from different modules;

- goals with varying costs / rewards;

- reducing uncertainty versus accomplishing goals.



Speech recognition with Sphinx


Robot control under uncertainty

Belief StateP(st=weather-today)=0.5

P(st=appointment-today )=0.5

USER

Action={ say-weather,update-appointment,clarify-query}

Speech=“today”

Stateweather-today


Robot control using Partially Observable MarkovPartially Observable Markov Decision Processes Decision Processes (POMDPs)

Belief state

USER + ENVIRONMENT + WORLD

Actions

ObservationsCosts / Rewards

State

Problem: Which action allows the robot to maximize its reward?

P(s1)P(s2)


Methods to solve POMDPs

Objective: Find a policy, (b), which maximizes reward.

Complexity

Performance

QMDP

MDP

FIB

UMDP

AMDP

O(S2A) O(S2AT) O(S2AO )O(S2AO) O(S2AB) T

POMDP

New methods?


New approach: A hierarchy of POMDPs

Idea: Exploit domain knowledge to divide one POMDP into many smaller ones.

Motivation: Complexity of POMDP solving grows exponentially with # of actions.

Assumption: We are given POMDP M = {S,A,,b,T,O,R} and hierarchy H

Act

ExamineHealth Navigate

MoveVerifyPulse

ClarifyGoal

North South East West

VerifyMeds

subtask

abstract action

primitive action


PolCA: Planning with a hierarchy of POMDPs

Step 1: Select the action setNavigate

Move ClarifyGoal

South East WestNorth

AMove = {N,S,E,W}

ACTIONSNorthSouthEastWest

ClarifyGoalVerifyPulseVerifyMeds





Step 1: Select the action set

Step 2: Minimize the state set

STATE FEATURESX-positionY-position

X-goalY-goal

HealthStatus


X-goalY-goal

HealthStatus

Navigate

Move ClarifyGoal


AMove = {N,S,E,W}

SMove = {X,Y}









Step 3: Choose parameters


X-goalY-goal

HealthStatus


X-goalY-goal

HealthStatus

Navigate

Move ClarifyGoal


AMove = {N,S,E,W}

SMove = {X,Y}





PARAMETERS

{bh,Th,Oh,Rh}

PARAMETERS

{bh,Th,Oh,Rh}





Step 3: Choose parameters

Step 4: Plan task h


X-goalY-goal

HealthStatus


X-goalY-goal

HealthStatus

Navigate

Move ClarifyGoal


AMove = {N,S,E,W}

SMove = {X,Y}





PLAN

h

PLAN

h

PARAMETERS

{bh,Th,Oh,Rh}

PARAMETERS

{bh,Th,Oh,Rh}


First study in simulation

• 20-questions domain:

Agent: “Is it an animal?”User: “No.”Agent: “Is it a vegetable?”User: “Yes.”Agent: “Is it green?”User: “No.”Agent: “...?”

Agent: “Is it an animal?”User: “No.”Agent: “Is it a vegetable?”User: “Yes.”Agent: “Is it green?”User: “No.”Agent: “...?”

Actions

Objective: Plan a sequence of questions allowing the agent to identify the chosen object.

Small complication…. the user can change objects at any time (Pr = 0.1), without telling the agent.

Observations


The hierarchy

Animals

Begin

Vegetable?Vegetables Minerals Mineral? Animal?

Mammal? RABBITHerbivore? TURTLE …?… …



Complexity

Performance

POMDP

QMDP

MDP

FIB

UMDP

AMDP

PolCA


Results

Domain: |S|=12, |A|=20, |O|=3

-120

-100

-80

-60

-40

-20

0

0.01 0.1 1 10 100 1000 10000 100000 1000000

Time (secs)

R

POMDP

PolCA-D1

FIB

QMDP


A new decomposition

Animals

Begin


… … …


A new decomposition

Fruits Plants

……

Animals

Begin


…

Fruit?

…


Results

-120

-100

-80

-60

-40

-20

0

0.01 0.1 1 10 100 1000 10000 100000 1000000

Time (secs)

R

POMDPPolCA-D1

PolCA-D2FIB

QMDP

Domain: |S|=12, |A|=20, |O|=3



Complexity

Performance

POMDP

QMDP

MDP

FIB

UMDP

AMDP

PolCA


PolCA in the Nursebot domain

• Goal: A robot is deployed in a nursing home, where it provides reminders to elderly users and accompanies them to appointments.

• Domain: |S|=512, |A|=20, |O|=19 • Hierarchy:


Sample scenario


Results for dialogue system

0.1 0.1

0.18

POMDP policy MDP policy


Summary

• We have developed a first prototype robot able to serve as a mobile nursing assistant for elderly people.

• The top-level controller uses a hierarchical variant of POMDPs to select actions.

• This allows it to acquire necessary information and successfully complete assigned tasks.

• Probabilistic techniques have been found to be very useful to flexibly model and track individuals.


For more details: www.cs.cmu.edu/~nursebot

The Nursebot team

CMU - Robotics:Greg Armstrong

Michael MontemerloJoelle PineauNicholas RoyJamie Schulte

Sebastian Thrun

CMU - HCI/Design:Francine Gemperle

Jennifer GoetzSarah KieslerAaron Powers

U. of Pittsburgh - Nursing:Jacqueline Dunbar-Jacobs

Sandra EngbergJudith Matthews

U. of Pittsburgh - CS:Don Chiarulli

Colleen McCarthy

U. of Freiburg - CS:Maren BennewitzWolfram Burgard

Dirk Schulz

U. of Michigan - CS:Laura BrownDirk ColbryCheryl OroszBart PeintnerMartha PollackSailesh Ramakrishnan

Standard Robotics:Greg Baltus


Our vision of robotic healthcare

Moving thingsaround

Moving thingsaround

Enabling use of remote

health services

Enabling use of remote

health services

Supportinginter-personal

communication

Supportinginter-personal

communication

Calling for helpin emergencies

Calling for helpin emergencies

Monitoring Rx adherence

& safety

Monitoring Rx adherence

& safety

Providinginformation

(TV, weather)

Providinginformation

(TV, weather)

Management support of

ADLs

Management support of

ADLsReminding

to eat, drink, & take meds

Reminding to eat, drink, & take meds

Providing physical

assistance

Providing physical

assistance

Linking the caregiver to resources

Linking the caregiver to resources


Profile of an aging population

450,000 nurses to recruit before 2008 (USA)


Localization and map building


People tracking


Analysis of movements


Autominder System


The family of Markov models

Markov Chain Hidden Markov Model(HMM)

Markov DecisionProcess (MDP)

Partially ObservableMarkov Decision

Process (POMDP)

State ambiguity?

no yes

Choice ofaction?

yes

no


The POMDP is a septuple { S, A, , b, T, O, R }

Problem: Which action allows the robot to maximize its reward?

bt-1 bt

at-1

ot

st-1 st

...

...

??ot-1

...

...

rt-1 rt

The POMDP model

Belief:

State:

Action:


Execution with a hierarchy of POMDPs

At each time step, traverse the hierarchy from top to bottom.

For each subtask, consult the policy at h(bt)

If at is an internal node > move to that subtask.

Act

ExamineHealth Navigate

MoveVerifyPulse

ClarifyGoal

North South East West

VerifyMeds


How to choose parameters?

• Consider for example TNavigate(s,a,s’)

• Case #1: a is a primitive action.

TNavigate(s,ClarifyGoal,s’) T(s,ClarifyGoal,s’)

Navigate

Move ClarifyGoal


S

S’South

S’

North

S’

S’

ClarifyGoal

0.7

0.3


How to choose parameters?

• Consider for example TNavigate(s,a,s’)

• Case #1: a is a primitive action.

TNavigate(s,ClarifyGoal,s’) T(s,ClarifyGoal,s’)

• Case #2: a is an abstract action.

TNavigate(s,Move,s’) T(s, Move(s) ,s’)

Navigate

Move ClarifyGoal


S

S’South

S’

North

0.9

0.8

0.1

0.2

S’

S’

ClarifyGoal


States: 1 per object (e.g. tomato, cucumber, rabbit, turtle, tulip, ….)

Observations: “yes”, “no”, “”

Actions: 1 guess per objet Rewards: question = -1+ many questions correct guess = +5

incorrect guess = -20

The details

Animals

Begin


Mammal? RABBITHerbivore? TURTLE …?… …

Towards robotic assistants in nursing homes: challenges and results

Documents

Transcript of Towards robotic assistants in nursing homes: challenges and results