Assisted Cognition

Henry KautzUniversity of Rochester

Computer Science

Vision

Understanding human behavior from sensor dataActively prompting, warning,and advisingAlerting caregivers as necessary

Computer systems that improve the independence and safety of people suffering from cognitive limitations by…

Technical Foundations

Inexpensive, easily deployed sensors

GPS cell phones – locationRFID tags – object manipulationWearables – motion, sound, pulse, ...

Advances in algorithms for high-level analysis of sensor data

General Approach

userprofile

generalknowledge

wearablesensors

environmentalsensors

decisionmaking

userinterface

caregiveralerts

behavior

cognitive state

intentions

activity

ExamplesMaintaining a daily schedule

Compensating for memory problemsCompensating for lowered self-initiative

NavigationIndoors or outdoors

Performing activities of daily livingStep-by-step prompting

Behavior regulationImproving self-awareness

Safety and wellnessNeed for immediate helpLong term health trends

Where is the Work Being Done?

Field is coming together from many communities

Computer science – artificial intelligence, robotics, ubiquitous computingRehabilitation engineeringRehabilitation medicineAssisted-living & nursing homesGerontology

ACCESS

Assisted Cognition in Community, Employment, & Social Settings

Example: Community Access for the Cognitively

Disabled

Problems in Using Public Transportation

Learning bus routes and numbersTransfers, complex plansRecovering from mistakes

Current GPS Navigation Devices

Designed for drivers, not bus riders!Should I get on this bus?Is my stop next?What do I do if I miss my stop?

Destination manually enteredHigh cognitive overhead

Device decides which route is “best”Familiar route better than shorter one

Catastrophic failure when signal is lost

Frequent “dead zones” in urban areas

Idea User carries GPS cell phoneSystem infers user’s state

Walking? Getting on a bus?

System learns about userImportant places, routes

Breaks from routine = user may be confused or lost

Offer helpCall caregiver

Transportation Routines

BA

Goal: intended destinationWorkplace, home, friends, restaurants, …

Trip segments: <start, end, mode>

Home to Bus stop A on Foot

Bus stop A to Bus stop B on Bus

Bus stop B to workplace on Foot

WorkplaceHome

GPS readingzk-1 zk

Edge, velocity, positionxk-1 xk

k-1 k Data (edge) association

Time k-1 Time k

mk-1 mk Transportation mode

tk-1 tk Trip segment

gk-1 gk Goal

Cognitive mode { routine, novel, error }

Dynamic Bayesian Network

ck-1 ck

Error Detectio

n: Missed

Bus Stop

GPS camera-phone“Knocks” when there is an opportunity to help

Can I guide you to a likely destination?I think you made a mistake!This place seems important – would you photograph it?

Prototype: Opportunity Knocks

CARE

Cognitive Assistance in Real-world Environments

GoalA home monitoring system that

Assists user in performing activities of daily livingTracks activities, and provides prompts and warnings as neededCan be deployed in an ordinary home

Initial Application

Accurate, automated ADL logs

Changes in routine often precursor to illness, accidentsHuman monitoring intrusive & inaccurate

Image Courtesy Intel Research

Technical RequirementsSensor hardware that can be practically deployed in a ordinary homeMethods for activity tracking from sensor data Methods for automated prompting that consider

Probability of user errorsProbability of system errorsCost / benefit tradeoffs

Object-Based Activity Recognition

Activities of daily living involve the manipulation of many physical objects

Kitchen: stove, pans, dishes, …Bathroom: toothbrush, shampoo, towel, …Bedroom: linen, dresser, clock, clothing, …

We can recognize activities from a time-sequence of object touches

Sensing Object Manipulation

RFID: Radio-frequency identification tags

SmallSemi-passiveDurableCheap

Near future: use products’ own tags

Wearable RFID Readers

Designed by Intel Research Seattle, samples given to a few academic partners

Hidden Markov Model

Trained on labeled data10-fold cross validation88% accuracy

Improving RobustnessTracking fails if novel (but reasonable) objects are used

Solution: smooth parameters over abstraction hierarchy of object types

Experiment: ADL Form Filling

Tagged real home with 108 tags14 subjects each performed 12 of 14 ADLs in arbitrary orderGiven trace, recreate activities

Results: Detecting ADLs

Activity Prior Work

SHARP

Personal Appearance 92/92

Oral Hygiene 70/78

Toileting 73/73

Washing up 100/33

Appliance Use 100/75

Use of Heating 84/78

Care of clothes and linen 100/73

Making a snack 100/78

Making a drink 75/60

Use of phone 64/64

Leisure Activity 100/79

Infant Care 100/58

Medication Taking 100/93

Housework 100/82

Legend

Point solution

General solution

Inferring ADLs from Interactions with Objects

Philipose, Fishkin, Perkowitz, Patterson, Hähnel, Fox, and Kautz

IEEE Pervasive Computing, 4(3), 2004

RFID

Current / Future DirectionsMore detailed sensor data

Machine vision fused with direct sensing

More detailed models of cognitive stateAffect recognition

Automated inventionsPrompting & guidanceInteraction: Audio? Visual? Tactile?

Interactive design/testing with various target populations

TBIAlzheimersAutism