Spatio-Temporal and Context Reasoning

in Smart Homes

Sook-Ling (Linda) Chua

Stephen Marsland, Hans W. Guesgen

School of Engineering and Advanced TechnologyMassey University, New Zealand

COSIT 2009

T

The world is aging

- have we noticed?

Source: United Nations (2007)

h e S i t u a t i o n . . .

The populations of the world are aging

Source: United Nations (2007)

T h e S i t u a t i o n . . .

The populations of the world are aging

Source: United Nations (2007)

T h e S i t u a t i o n . . .

The populations of the world are aging

Source: United Nations (2007)

T

< 10%

> 25%

h e S i t u a t i o n . . .

T People choose to stay in their own homes

as long as possible and remain independent

h e S i t u a t i o n . . .

T

Aging

Physical disability Cognitive impairment

diminished sense and touch slower ability to react poor vision, hearing problems memory problems

leads to

People choose to stay in their own homes

as long as possible and remain independent

h e S i t u a t i o n . . .

Supporting inhabitant’s daily activities

T h e S i t u a t i o n . . .

“Smart Homes”

Figure extracted from: http://www.dreamhomesmagazine.com/

T h e S i t u a t i o n . . .

To react intelligently, the smart home needs to:

(1) recognise inhabitant’s behaviour

(2) perform reasoning

spatio-temporal information

contextual information

sensor output

“Tokens”

B

Figure extracted from: http://www.dreamhomesmagazine.com/

The Smart Home

e h a v i o u r R e c o g n i t i o n

“Tokens” The direct representation of current sensor states being triggered

E.g. of a sequence of tokens from the sensors

Date Activation Time Activation Room Object Type Sensor State

16/6/2008 18:05:23 Living room Television Off

16/6/2008 18:08:19 Living room Curtain Closed

16/6/2008 18:09:48 Kitchen Light On

16/6/2008 18:10:35 Kitchen Cupboard Open

16/6/2008 18:25:06 Kitchen Fridge Open

16/6/2008 19:00:02 Laundry Washing Machine

On

B e h a v i o u r R e c o g n i t i o n

“Tokens” The direct representation of current sensor states being triggered

E.g. of a sequence of tokens from the sensors

Date Activation Time Activation Room Object Type Sensor State

16/6/2008 18:05:23 Living room Television Off

16/6/2008 18:08:19 Living room Curtain Closed

16/6/2008 18:09:48 Kitchen Light On

16/6/2008 18:10:35 Kitchen Cupboard Open

16/6/2008 18:25:06 Kitchen Fridge Open

16/6/2008 19:00:02 Laundry Washing Machine

On

B e h a v i o u r R e c o g n i t i o n

12S19th Jan 2009 18:03:16

4S19th Jan 2009 18:07:56

78S19th Jan 2009 18:20:27

101M19th Jan 2009 18:33:44

11S19th Jan 2009 18:50:12

23S19th Jan 2009 19:01:08

5S

19th Jan 2009 19:37:2117M

19th Jan 2009 19:41:26

.

.

.

.

.

.

…

KitchenDining/Living

Room

Bedroom

Laundry

Office/Study

Figure extracted from: The Aware Home, 2002

B e h a v i o u r R e c o g n i t i o n

12S19th Jan 2009 18:03:16

4S19th Jan 2009 18:07:56

78S19th Jan 2009 18:20:27

101M19th Jan 2009 18:33:44

11S19th Jan 2009 18:50:12

23S19th Jan 2009 19:01:08

5S

19th Jan 2009 19:37:2117M

19th Jan 2009 19:41:26

.

.

.

.

.

.

…

KitchenDining/Living

Room

Bedroom

Laundry

Office/Study

Figure extracted from: The Aware Home, 2002

B e h a v i o u r R e c o g n i t i o n

12S19th Jan 2009 18:03:16

4S19th Jan 2009 18:07:56

78S19th Jan 2009 18:20:27

101M19th Jan 2009 18:33:44

11S19th Jan 2009 18:50:12

23S19th Jan 2009 19:01:08

19th Jan 2009 19:37:2117M

19th Jan 2009 19:41:26

.

.

.

.

.

.

…

KitchenDining/Living

Room

Bedroom

Laundry

Office/Study

Q: How do we recognise behaviours?

Figure extracted from: The Aware Home, 2002

B e h a v i o u r R e c o g n i t i o n

B e h a v i o u r R e c o g n i t i o n

KitchenDining/Living

Room

Bedroom

Laundry

Office/Study

12S19th Jan 2009 18:03:16

4S19th Jan 2009 18:07:56

78S19th Jan 2009 18:20:27

101M19th Jan 2009 18:33:44

11S19th Jan 2009 18:50:12

23S19th Jan 2009 19:01:08

19th Jan 2009 19:37:21 17M...

.

.

.

Figure extracted from: The Aware Home, 2002

Challenges:

(a) Exact activities are not directly observed, only the

sensor observations

. . .

B e h a v i o u r R e c o g n i t i o n

KitchenDining/Living

Room

Bedroom

Laundry

Office/Study

12S19th Jan 2009 18:03:16

4S19th Jan 2009 18:07:56

78S19th Jan 2009 18:20:27

101M19th Jan 2009 18:33:44

11S19th Jan 2009 18:50:12

23S19th Jan 2009 19:01:08

19th Jan 2009 19:37:21 17M...

.

.

.

. . .

Figure extracted from: The Aware Home, 2002

Challenges:

(a) Exact activities are not directly observed, only the

sensor observations

B e h a v i o u r R e c o g n i t i o n

KitchenDining/Living

Room

Bedroom

Laundry

Office/Study

12S19th Jan 2009 18:03:16

4S19th Jan 2009 18:07:56

78S19th Jan 2009 18:20:27

101M19th Jan 2009 18:33:44

11S19th Jan 2009 18:50:12

23S19th Jan 2009 19:01:08

19th Jan 2009 19:37:21 17M...

.

.

.

Figure extracted from: The Aware Home, 2002

Challenges:

(a) Exact activities are not directly observed, only the

sensor observations

?

. . .

B e h a v i o u r R e c o g n i t i o n

Challenges:

(b) Same sensor activations will be involved in

multiple behaviours

B e h a v i o u r R e c o g n i t i o n

Challenges:

(b) Same sensor activations will be involved in

multiple behaviours

Figure extracted from: www.rebecca-waring.com, www.cyh.com, www.chow.com

B e h a v i o u r R e c o g n i t i o n

Challenges:

(c) No. of observations can vary between activities

Making breakfast Making dinner

Fridge

Toaster

Cupboard Cupboard

Stove

Microwave Oven

Tap

Drawer

B e h a v i o u r R e c o g n i t i o n

Challenges:(d) Behaviours are rarely identical on each use

E.g. Making a cup of tea

With / without Milk / water first? How long?

components can be present/absent the order of individual components happen can change length of time each piece takes can change

B e h a v i o u r R e c o g n i t i o n

Challenges:(d) Behaviours are rarely identical on each use

E.g. Making a cup of tea

With / without Milk / water first? How long?

components can be present/absent the order of individual components happen can change length of time each piece takes can change

Stochastic Approach

B e h a v i o u r R e c o g n i t i o n

The Hidden Markov Model (HMM)

probabilistic graphical model

Source: Rabiner, L. (1989)

uses probability distributions to determine the

states for a sequence of observations over time

B e h a v i o u r R e c o g n i t i o n

The Hidden Markov Model (HMM)

probabilistic graphical model

Source: Rabiner, L. (1989)

1tO tO 1tOObservations We know this..

uses probability distributions to determine the

states for a sequence of observations over time

B e h a v i o u r R e c o g n i t i o n

The Hidden Markov Model (HMM)

probabilistic graphical model

Source: Rabiner, L. (1989)

1tO tO 1tOObservations We know this..

1tS tS 1tS ……States But, not this

uses probability distributions to determine the

states for a sequence of observations over time

1tS tS 1tS

1tO tO 1tO

…

…States

Observations

Markov property:

The probability of transition to a state (St+1) depends only

on the current state (St) [represented by solid line]

The observation at Ot depends only on the state St

at that time slice [represented by dashed line]

B e h a v i o u r R e c o g n i t i o n

…

The Hidden Markov Model (HMM)

Source: Rabiner, L. (1989)

1tS tS 1tS

1tO tO 1tO

…

…States

Observations

Markov property:

The probability of transition to a state (St+1) depends only

on the current state (St) [represented by solid line]

The observation at Ot depends only on the state St

at that time slice [represented by dashed line]

B e h a v i o u r R e c o g n i t i o n

…

The Hidden Markov Model (HMM)

Source: Rabiner, L. (1989)

1tS tS 1tS

1tO tO 1tO

…

…States

Observations

Markov property:

The probability of transition to a state (St+1) depends only

on the current state (St) [represented by solid line]

The observation at Ot depends only on the state St

at that time slice [represented by dashed line]

B e h a v i o u r R e c o g n i t i o n

…

The Hidden Markov Model (HMM)

Source: Rabiner, L. (1989)

B e h a v i o u r R e c o g n i t i o n

KitchenDining/Living

Room

Bedroom

Laundry

Office/Study12S19th Jan 2009 18:03:16

4S19th Jan 2009 18:07:56

78S19th Jan 2009 18:20:27

101M19th Jan 2009 18:33:44

11S19th Jan 2009 18:50:12

23S19th Jan 2009 19:01:08...

.

.

.

B e h a v i o u r R e c o g n i t i o n

KitchenDining/Living

Room

Bedroom

Laundry

Office/Study12S19th Jan 2009 18:03:16

4S19th Jan 2009 18:07:56

78S19th Jan 2009 18:20:27

101M19th Jan 2009 18:33:44

11S19th Jan 2009 18:50:12

23S19th Jan 2009 19:01:08...

.

.

.

4S12S 78S101MObservations

B e h a v i o u r R e c o g n i t i o n

KitchenDining/Living

Room

Bedroom

Laundry

Office/Study12S19th Jan 2009 18:03:16

4S19th Jan 2009 18:07:56

78S19th Jan 2009 18:20:27

101M19th Jan 2009 18:33:44

11S19th Jan 2009 18:50:12

23S19th Jan 2009 19:01:08...

.

.

.

?

4S12S 78S101MObservations

Cupboard CoffeeMachine

FridgeStates

To use HMM to recognise behaviours:

(1) Segmentation break the token sequence into

appropriate pieces that represent individual

behaviours

1 32 54 6 T. . .7 Observations

start startend end

B e h a v i o u r R e c o g n i t i o n

1 32 54 6 T. . .7

(2) Classification identify the behaviours using

the HMM

Observations

To use HMM to recognise behaviours:

“Behaviour A” “Behaviour B”

B e h a v i o u r R e c o g n i t i o n

Behaviour Recognition using HMM

Our approach:

Use a set of HMMs that each recognise different

behaviours

“Making coffee” “Showering”

These HMMs will compete to explain the current

observations Model selection is based on maximum likelihood

“Making lunch”

. . .

B e h a v i o u r R e c o g n i t i o n

Source: Chua, Marsland and Guesgen (2009)

Experiment: Competition between HMMs

Datasets

MIT PlaceLab Designed a set of simply installed state-change

sensors that were placed in two different apartments

with real people living in them

Source: Tapia (2004)

B e h a v i o u r R e c o g n i t i o n

Experiment: Competition between HMMs

Datasets

The subjects kept a record of their activities that form a

set of annotations for the data

“Ground-truth” segmentation of the dataset

We used the dataset from the first subject

77 sensors collected for 16 consecutive days

B e h a v i o u r R e c o g n i t i o n

Datasets

Activities take place in one room (kitchen) Location of the sensors is known a priori Behaviours:

Prepare breakfast (toaster) Prepare breakfast (cereal) Prepare beverage Prepare lunch Do the laundry

Experiment: Competition between HMMs

B e h a v i o u r R e c o g n i t i o n

Based on 727 observations (using 11 days testing and 5 days training set)

B e h a v i o u r R e c o g n i t i o n

Based on 727 observations (using 11 days testing and 5 days training set)

B e h a v i o u r R e c o g n i t i o n

Based on 727 observations (using 11 days testing and 5 days training set)

B e h a v i o u r R e c o g n i t i o n

Experimental Results

Method works effectively

performs segmentation and detects changes of activities

B e h a v i o u r R e c o g n i t i o n

Microwave Fridge Coffee Machine

DrawerDrawer

1 2 3 4 5 6 7 . . .observation

T

Fridge Cupboard

Experimental Results

Method works effectively

performs segmentation and detects changes of activities

B

Preparing lunch Preparing a beverage

Microwave Fridge Coffee Machine

DrawerDrawer

1 2 3 4 5 6 7 . . .observation

T

Fridge Cupboard

e h a v i o u r R e c o g n i t i o n

Discussion

Lack of spatio-temporal information

Misclassification:

The end of one behaviour contains observations that

should be the start of the next

Microwave Cupboard Fridge Fridge Coffee Machine DrawerDrawer1 2 3 4 5 6 7 …

observation

Preparing lunch Preparing a beverage

T

B e h a v i o u r R e c o g n i t i o n

Discussion

Lack of spatio-temporal information

Misclassification:

The end of one behaviour contains observations that

should be the start of the next

Microwave Cupboard Fridge Fridge Coffee Machine DrawerDrawer1 2 3 4 5 6 7 …

observation

Preparing lunch Preparing a beverage

T

B e h a v i o u r R e c o g n i t i o n

Discussion

Lack of spatio-temporal information

Misclassification:

The end of one behaviour contains observations that

should be the start of the next

Microwave Cupboard Fridge Fridge Coffee Machine DrawerDrawer1 2 3 4 5 6 7 …

observation

Preparing lunch Preparing a beverage

T

B e h a v i o u r R e c o g n i t i o n

Microwave Cupboard Fridge Fridge Coffee Machine DrawerDrawer1 2 3 4 5 6 7 . . .

observation

Preparing lunch Preparing a beverage

T

Preparing lunch Preparing a beverage

Discussion

Lack of spatio-temporal information

Misclassification:

The end of one behaviour contains observations that

should be the start of the next

B e h a v i o u r R e c o g n i t i o n

A: Augment current algorithm to include spatio-temporal information

Q: How to reduce misclassification?

Discussion

Lack of spatio-temporal information

B e h a v i o u r R e c o g n i t i o n

NOT directly interested in the exact coordinates

Spatial information (Where?)

S

So, what are we interested in?

Room location

e.g.

Figures extracted from: www.istockphoto.com, www.clubjam.jammag.com, www.nancilea.blogspot.com

p a t i o - t e m p o r a l

Spatial information (Where?)

S p a t i o - t e m p o r a l

Current study used very basic spatial information

(just the kitchen!)

In the future, . . .

B e d r o o m K i t c h e n

B a t h r o o mD i n i n g R o o m

L i v i n g R o o m

Preparing a beverage

Cooking

Showering

Grooming

Computing

Sleeping

Washing dishes

Eating

Reading

Watching TV

Exercising

Sitting around fireplace

Resting

S p a t i o - t e m p o r a l

B e d r o o m K i t c h e n

B a t h r o o mD i n i n g R o o m

L i v i n g R o o m

Preparing a beverage

Cooking

Showering

Grooming

Computing

Sleeping

Washing dishes

Eating

Reading

Watching TV

Exercising

Sitting around fireplace

Resting

S p a t i o - t e m p o r a l

S p a t i o - t e m p o r a l

. . .

observation

TMicrowave Cupboard FridgeDrawer

1 2 3 4

Preparing lunch

Fridge Coffee Machine

Drawer7 8 9

Preparing a beverage

Fan5

Shower6

Showering

Kitchen Bathroom Kitchen

Spatial information (Where?)

S p a t i o - t e m p o r a l

Spatial information (Where?)

. . . is this sufficient for reasoning?

WITHOUT temporal, the system cannot differentiate:

Bathroom

3 am

8 am

Vs.

Figure extracted from: http://hazard.com/graphics

S p a t i o - t e m p o r a l

Temporal information

When does a behaviour occur?

Source: Guesgen and Marsland (2009)

How long does behaviour take?

How often does behaviour occur?

Mapping to time scale

Duration

Frequency

e.g. Mary vacuums every Sunday

e.g. Microwave used for a dangerously long time

e.g. Peter showers 3 times a day

S p a t i o - t e m p o r a l

Temporal information (When)

3.03 pm

weekends vs. weekdays

winter vs. summer

½ hour after shower

having breakfast 2 hours before meeting

am vs. pm

Absolute time Relative time

.

.

.

.

S p a t i o - t e m p o r a l

Time scales

Yearly (e.g. Christmas, New Year, Easter, etc.)

Weekly (e.g. vacuuming, visit from health worker, etc.)

Daily (e.g. showering, eating, etc.)

S p a t i o - t e m p o r a l

Temporal information

(a) segment the behaviours

(b) generate a sequence of behavioural patterns

tells us when, for how long and how frequent behaviour occurs

Source: Guesgen and Marsland (2009)

S p a t i o - t e m p o r a l

a m p m

E v e n i n g

N i g h t

Preparing a beverage

Cooking

Showering

Grooming Computing

Washing dishes

Eating

Reading

Watching TV

Exercising

Resting

Sleeping

Watching TVCooking

Cooking

Washing dishes

Washing dishes

Computing

Computing

Preparing a beverage

Sitting around fireplace

Exercising

Reading

Reading

Eating

Eating

Preparing a beverage

Showering

S p a t i o - t e m p o r a l

a m p m

E v e n i n g

N i g h t

Preparing a beverage

Cooking

Showering

Grooming Computing

Washing dishes

Eating

Reading

Watching TV

Exercising

Resting

Sleeping

Watching TVCooking

Cooking

Washing dishes

Washing dishes

Computing

Computing

Preparing a beverage

Sitting around fireplace

Exercising

Reading

Reading

Eating

Eating

Preparing a beverage

Showering

S p a t i o - t e m p o r a l

a m p m

E v e n i n g

N i g h t

Preparing a beverage

Cooking

Showering

Grooming Computing

Washing dishes

Eating

Reading

Watching TV

Exercising

Resting

Sleeping

Watching TVCooking

Cooking

Washing dishes

Washing dishes

Computing

Computing

Preparing a beverage

Sitting around fireplace

Exercising

Reading

Reading

Eating

Eating

Preparing a beverage

Showering

S p a t i o - t e m p o r a l

Temporal information

(a) segment the behaviours

(b) generate a sequence of behavioural patterns

tells us when, for how long and how frequent behaviour occurs

S p a t i o - t e m p o r a l

Time

Cooking

Reading

Resting

Preparing a beverage

Washing dishes

Eating

Watching TV

Computing

Exercising

Cooking

Watching TV

Behaviour

Time

Cooking

Reading

Resting

Preparing a beverage

Washing dishes

Eating

Watching TV

Computing

Exercising

Cooking

Watching TV

.

.

.

Behaviour

S p a t i o - t e m p o r a l

Dining Room

Eating

Reading

Kitchen

Preparing a beverage

Cooking

Washing dishes

Living Room

Watching TV

Exercising

Sitting around fireplace

Resting

Space

Competition among HMMs

Bedroom

Grooming

Computing

Sleeping

Time

Cooking

Reading

Resting

Preparing a beverage

Washing dishes

Eating

Watching TV

Computing

Exercising

Cooking

Watching TV

Behaviour

S p a t i o - t e m p o r a l

Space

Competition among HMMs

.

.

.

Dining Room

Eating

Reading

Kitchen

Preparing a beverage

Cooking

Washing dishes

Living Room

Watching TV

Exercising

Sitting around fireplace

Resting

Bedroom

Grooming

Computing

Sleeping

Time

Cooking

Reading

Resting

Preparing a beverage

Washing dishes

Eating

Watching TV

Computing

Exercising

Cooking

Watching TV

Behaviour

S p a t i o - t e m p o r a l

Space

Competition among HMMs

.

.

.

Dining Room

Eating

Reading

Kitchen

Preparing a beverage

Cooking

Washing dishes

Living Room

Watching TV

Exercising

Sitting around fireplace

Resting

Bedroom

Grooming

Computing

Sleeping

Time

Cooking

Reading

Resting

Preparing a beverage

Washing dishes

Eating

Watching TV

Computing

Exercising

Resting

Watching TV

Behaviour

S p a t i o - t e m p o r a l

Space

Competition among HMMs

.

.

.

Dining Room

Eating

Reading

Kitchen

Preparing a beverage

Cooking

Washing dishes

Living Room

Watching TV

Exercising

Sitting around fireplace

Resting

Bedroom

Grooming

Computing

Sleeping

What happens if the person is late one day

and makes lunch at 3 pm?

The system may make mistakes, particularly

with time!

Fuzzy logic system

S p a t i o - t e m p o r a l

C

How was the current situation is reached?

Contextual information

What else is happening?

What is the state of the environment?

. . . needs to be considered!

o n t e x t u a l R e a s o n i n g

C

. . . is this normal?

“ John is boiling water in the middle of the night ”

o n t e x t u a l R e a s o n i n g

C

“ John is boiling water in the middle of the night ”

Spatial: Kitchen

Temporal: Middle of the night

Is the information sufficient for reasoning?

o n t e x t u a l R e a s o n i n g

C

“ John is boiling water in the middle of the night

after watching late night movie ”

Contextual information

Spatial: Living room Kitchen

Temporal: Middle of the night and is Saturday

. . . he stays up longer !!!

o n t e x t u a l R e a s o n i n g

C

Competition between HMMs a possible mechanism for behaviour recognition and segmentation

Spatio-temporal and context awareness play an important role in interpreting behaviour

o n c l u s i o n

A

Stephen Marsland, Hans Guesgen

Massey University Smart Environment

(MUSE) members

School of Engineering and Advanced

Technology (SEAT)

Massey University

c k n o w l e d g e m e n t s

Thank you!(Merçi!)

F i n a l l y . . .