Semantic Reasoning and Verification for Ambient Assisted Living

Thibaut Tiberghien, Mounir Mokhtari, Liu Yan & Dong Jin Song !!

FSFMA 2014 2nd French Singaporean Workshop on Formal Methods and Applications

13 May 2014, Singapore

OutlineI. Introduction

II. Context reasoning

- Reasoning strategies

- Reasoning architecture

III. Validation & Verification

IV.Conclusion

!2

- Major demographic changes *

Population ageing

2010

10

20

30

40

50

60

70

80

over 85Males Females

0

2050

10

20

30

40

50

60

70

80

over 85Males Females

00% 2% 4% 6%2%4%6% 0% 2% 4% 6%2%4%6%

Age Age

!3* United Nations, World Population Prospects: The 2010 Revision. http://esa.un.org/unpd/ wpp/index.htm

- Elderly support ratio is shrinking

- e.g. in Singapore *

Population ageing

From 13.5 in 1970 to 2.6 in 2015!

!4* Resident Old-Age Support Ratio. Department of Statistics Singapore. http://www.singstat.gov.sg

Progression of dementia- Projections of frail elderly population in Singapore *

!5* Dr. Philip Yap, Khoo Teck Puat Hospital, Singapore

Num

ber o

f fra

il el

derly

5000

15000

25000

35000

45000

2000 2010 2020 2030

7 000 beds available in nursing homes Need in 2015: 22 000 beds

Sensing Context-aware framework Service provision

What is AAL?

!6

Scientific challenges

!7

Sensing Context-aware framework Service provision

« Sensor prototyping »

« Micro context acquisition »

« Context awareness »

« Reasoning techniques »

« Modelling » « Service platform »

« User interface plasticity »

« Human factors »

Communication layer

System verification« Rules verification »

« Information consistency »

« Wireless communication »« Discovery protocols »

« Privacy »

Scientific challenges

!8

Sensing Context-aware framework Service provision

« Sensor prototyping »

« Micro context acquisition »

« Context awareness »

« Reasoning techniques »

« Modelling » « Service platform »« User interface

plasticity »

« Human factors »

System verification« Rules verification »

« Information consistency »

« Wireless communication »« Discovery protocols »

« Privacy »

collaboration with Liu Yan, Dong Jin Song, NUS (PAT)

our research work

Communication layer

OutlineI. Introduction

II. Context reasoning

- Reasoning strategies

- Reasoning architecture

III. Validation & Verification

IV.Conclusion

!9

Related work

!10

Rule-based (Semantic Web)

Pattern matching (HMM, DBN…)

Sensor fusion and

binding to context

Detection of patterns in sequence of

preprocessed sensor data

+ no prior knowledge

data driven !-

labelled dataset for learning fine grain sensing

+ leverage prior knowledge

no learning !-

no implicit patterns sensitive to noise

Semantic Web Technologies- Metadata-data graph model

- Make statements about resources

- Triple-based representation

!

- URI and namespaces

- RDF, RDF-S, OWL (TBox)

- Own namespaces (TBox + Abox)

- Serializations: XML, N3, ...

!11

« subject predicate object »

Functional context model

!12Legend

datatypeProperty

Class

objectProperty

subClassOf

stageForAlertisAloneinRoomSinceinRoomFordoesActivitySincedoesActivityFor

Resident

liveIn/detectedIn/cameFrom

motionMeasured

Environment

lastUsed

Object

hasValuelastUpdatehasLastUpdate

Sensor

indicateLocationindicateUse

SensorState

Furniture

Door

SensorType

HouseRoomOutside

BedroomBathroom

LivingroomKitchenToilet

...

useNowlocatedIn

partOf

type

hasPossibleState/hasCurrentState

attachedTo

deployedIn

namesnoozeTime

believedToDo

nameneedHandsgetConfidenceScore

Context

busy

Caregiver

LocationActivity

solvedriskLevel

Deviance

Reminder

ackHandledacknowledgement

Notification

namemodelrepeatidtimeSentstage

Service

name

Device

watchesAfter

runningFor

solvedBy

helpsWith

usedBy

escalateTo

hasAckService

deployedIn

onDevice

Person

Rule design: epistemology- Type I: Rules of reasoning

- Irrevocable knowledge

- Scientific models, common sense

- Type II: Rules of fact

- Domain knowledge

- Experts’ experience, intuition and insight

- Possible contradiction (scoring)

- Type III: Arbitration rules

!13

Type I - Rules of reasoning

!14

∀ Sensor se; SensorState st; Room r, r2; House h; Resident u; Timestamp t; !

{se hasCurrentState st} ⋀ {se hasLastUpdate true} ⋀ {st indicateLocation true} ⋀ {se deployedIn r} ⋀ {r partOf h} ⋀ {u liveIn h} ⋀ {u detectedIn r2} ⋀ {r2 ≠ r}

⋀ {se lastUpdate t} ⟹ {u detectedIn r} ⋀ {u cameFrom r2} ⋀ {u inRoomSince t}

Type II - Rules of fact

!15

Type III - Arbitration rules

Compute the Rule-Based Confidence Score

!16

OutlineI. Introduction

II. Context reasoning

- Reasoning strategies

- Reasoning architecture

III. Validation & Verification

IV.Conclusion

!17

« La Pensée »A Cognitively Inspired Hybrid Reasoning Architecture

!18

Cogitation (conscious mind)

Cerebration (unconscious mind)

Knowledge formalisation

Stimulus (~ event)

Cerebral cortex

Live event processing

!19

Legend

invokeservice

exchangedata

model.n3 home.n3 rules.n3 dump.n3

Stimulistener Cortex

Cerebration* Cogitation

EYE

DecisionInterpreter

NTriplestore

1.live event

0.loaded once when ubismart starts

2.update(live event)

3.think

i.think iv.think

ii.query iii.update v.dump

vii.think

x.update ix.inferred triples

vi.ontological dumpat each inference

viii.loaded in memoryat each inference

5.decision

xi.decision

Hybrid reasoning

!20

Service Delivery Modules

Sensing Modules

Configuration Modules

Reasoning Modules

"X"

SensorFlowManager (Routing)

- merge all "X" signals- registration of SensApps- forward data to relevant SensApp- forward information to InfoConsumers- discovery of new sensors- configuration gate for Binder- sensors to SensApps bindings

SensApp*

- process data- produce information

BinderUI*

SensAppRepository*

- list SensApps per category of sensor- provide SensApp for download

"X" Gateway*

- low level- protocol dependent (code is specific)- normalization of data (e.g. HEX2DEC)- signal forwarding

D I K W?

- flow- no meaning- mostly useless

- in the semsea (ontology)- semantic (meaning)- useful & calculable

SensApp dev.> introduce libs in SensApp API> restructure data in a more meaningful way using processes

INFERENCE> process is not necessary semantic but I/O goes through semantic sea

Binder

- configuration tool (build, declarative)- choice, download & installation of SensApps- parametrization of SensApps- bind semantics (to sensor + its events)

Cortex

- handle reasoning cycle (until stable cogitative state)- alternate between cogitation pass & cerebration pass

start/stop & feedback

REST + Event Bus

handleEvent

processEvent

processThoughts

REST + download

Legend

SmartSpaceComposer (S2C): configuration

* this module can have several instances this module is semantic-aware, i.e. linked with NTriplestore

External Module

Internal Module

Abstract Class

EventBridge*

XMPPBridge

DPWSBridge

HTTPBridge

OSGiBridge

publish

subscribe

Stimulistener

- match sensor/state with semsea bindings- write context information in semseasubscribe

NTriplestore

- API to semantic sea (semsea)- provide CRUD access to the triplestore- implemented as a service

think

Cogitation

- high level rule-based reasoning- decision making- conscious process

Cerebration*

- threaded asynchronous inference- low level, specific, computational task- I/O in seamsea- subconscious process

Classification

Statistical Analysis

Numeric Computation

CoreEngine*

EYE

SVM

think

DecisionInterpreter

- interpret thoughts (reasoning results)- update triplestore- start/stop service- etc.

ServiceGateway

- registration of services- control over services- status update in triplestore

start/stop/update

UbiService*

BackEndService*FrontEndService*

AtomicService* CoupledService*

Reminder* CallCenter*

ActivityLog*Notification*

display(<ServiceSkin>) & ack

register

DeviceGateway

- registration & management of devices- semsea I/O

register

ComLayer*

XMPP

APNs

UbiDevice*

IPhone*

IPTV*

send & ack

ServiceSkin*

MobileGUI*

TextToSpeech*

...

- fused information- domain knowledge- higher level

subscribe

InfoConsumer*

- platform or application using the same data

Ontology evolution breakdown- initial ontology

!21

Legend

object

datatype property

litteral

object

object property

update

query

johndoe

occupied livingroom

toilet

house

pir2

pir2_off

pir6

pir6_on

believedToDo detectedInpartOf

liveIn

deployedIn

deployedIn

hasCurrentState

hasCurrentState

motion

partOf

7

motion0

motion 7

id

A2

id

A6hasLastUpdate

2012-06-27 13:07:58

hasLastUpdate

2012-06-27 08:42:27

Legend

object

datatype property

litteral

object

object property

update

query

johndoe

occupied livingroom

toilet

house

pir2

pir2_on

pir2_off

pir6

pir6_on

believedToDo detectedInpartOf

liveIn

deployedIn

deployedIn

hasCurrentState

hasCurrentState

motion

partOf

7

motion0

motion 7

id

A2

id

A6hasLastUpdate

2012-06-27 13:07:58

hasLastUpdate

2012-06-27 13:08:08

hasCurrentState

Ontology evolution breakdown- Stimulistener: new event update

!22

Ontology evolution breakdown- Cerebration: motion estimation

!23

Legend

object

datatype property

litteral

object

object property

update

query

johndoe

occupied livingroom

toilet

house

pir2

pir2_onpir6

pir6_on

believedToDo detectedInpartOf

liveIn

deployedIn

deployedIn

hasCurrentState

motion

partOf

7

motion

1

motion 8

id

A2

id

A6hasLastUpdate

2012-06-27 13:07:58

hasLastUpdate

2012-06-27 13:08:08

hasCurrentState

Ontology evolution breakdown- Cogitation: activity inference

!24

{?u qol:detectedIn ?r. ?r a qol:Bathroom. ?u qol:inRoomFor ?d. ?d math:lessThan 1800. ?a1 a qol:Hygiene. ?u qol:canDo ?a1} => {?a1 qol:getScore 9}.

Legend

object

datatype property

litteral

object

object property

update

query

johndoe

occupied

hygiene

livingroom

toilet

house

pir2

pir2_onpir6

pir6_on

believedToDo cameFrompartOf

liveIn

deployedIn

deployedIn

hasCurrentState

motion

partOf

7

motion1

motion 8

id

A2

id

A6hasLastUpdate

2012-06-27 13:07:58

hasLastUpdate

2012-06-27 13:08:08

believedToDo detectedIn

hasCurrentState

OutlineI. Introduction

II. Context reasoning

- Reasoning strategies

- Reasoning architecture

III. Validation & Verification

IV.Conclusion

!25

Traditional AAL validation- Difficulty to obtain relevant ground truth

- Synthetic dataset

- Role-played and labelled dataset (living lab)

!

- Irrelevant to real settings: is system usable?

- Does not validate economical "performance"

- Does not certify system behaviour

!26

A 3-fold validation process- Performance: Load test

!

- Usability: In-situ system trial

- STARhome: controlled (technical validation)

- Singapore nursing home: semi-controlled (top-down)

- French individual homes: no control (bottom-up)

!

- Certification: Rules verification

!27

Performance- Reasoning load test to 250 houses (40K+ triples)

!28

0

10

20

30

40

50

60

1 (34

4)

16 (2

804)

31 (5

264)

46 (7

724)

61 (1

0184

)

76 (1

2644

)

91 (1

5104

)

106 (

1756

4)

121 (

2002

4)

136 (

2248

4)

151 (

2494

4)

166 (

2740

4)

181 (

2986

4)

196 (

3232

4)

211 (

3478

4)

226 (

3724

4)

241 (

3970

4)

EYE

Rea

soni

ng D

urat

ion

(s)

Nb of houses (Nb of triples)

34s

200 homes

STARhome

!29

Kitchen Livingroom Bedroom Bathroom

Pressure mat(ZigBee)

iPad(WiFi)

Ultrasound(ZigBee)

TV + Set top box(LAN)

Karotz(WiFi)

Karotz(WiFi)

Pressure mat(Web Service)

Water meter(Web Service)

Back-end

ZigBeegateway

Wi-Fi router

Tiny debian machine

Server with webservice repository

Serial/USBLAN

Nursing home in Singapore

!30

-14 months trial -8 residents with dementia -2 caregivers

Results- Recognition rate of events

!

!

!

- Early detection of the health issues

!31

events missed

correct recognition

events missed

correct recognition

Feb. 2012 (phase 2) Sep. 2012 (phase 3)

17%29%

71% 83%

0

3

6

9

12

24/02/12 27/02/12 01/03/12 04/03/12 07/03/12 10/03/12 13/03/12

Nb

of r

emin

ders

per

day

Individual homes in France

!32

Bedroom 1

Livingroom

Bedroom 2

Kitchen

WC

Bath

door open/close sensor

motion sensor

internet-connected gateway (BeagleBone)

- 3 homes - 2 months trial in average

Results- Activity inference

!33

Week 1

Week 2

Week 3

Week 4

Week 5

Week 6

00:00 06:00 12:00 18:00 00:00

Go out

Nap

Occupied

Sleep

time

day

Results- Contextual dashboard

!34

2012-06-04

640640640640640

2012-06-05

824824824824824

2012-06-06339339339339339

2012-06-07

11111

2012-06-08362362362362362

2012-06-09

781781781781781

2012-06-10 470470470470470

2012-06-11

543543543543543

2012-06-12

835835835835835

2012-06-13388388388388388

2012-06-14340340340340340

2012-06-15

585585585585585

2012-06-16362362362362362

2012-06-17371371371371371

2012-06-18

590590590590590

2012-06-19 409409409409409 2012-06-20

134134134134134

2012-06-21

274274274274274

2012-06-22

44444

2012-06-23 394394394394394 2012-06-24

256256256256256

2012-06-25

288288288288288

2012-06-26 471471471471471 2012-06-27 422422422422422 2012-06-28 446446446446446 2012-06-29518518518518518

2012-06-30 418418418418418 2012-07-01

191191191191191

2012-07-02

309309309309309

2012-07-03

250250250250250

2012-07-04

299299299299299

2012-07-05

11111

2012-07-06

11111

2012-07-07

11111

2012-07-08

11111

2012-07-09

4242424242

2012-07-10

309309309309309

2012-07-11

172172172172172

2012-07-12

199199199199199

Outside

Living room

Kitchen

Bedroom3

Bedroom2

Bedroom

Mon. Tue. Wed. Thu. Fri. Sat. Sun.

OutlineI. Introduction

II. Context reasoning

- Reasoning strategies

- Reasoning architecture

III. Validation & Verification

IV.Conclusion

!35

Liu Yan & Dong Jin Song (PAT)

@Peacehaven Nursing Home

Model checking of AAL system- People + environment + system modelling

- Formalisms for concurrent interaction

!

- Comprehensive verification for certification

- Automatic verification and exhaustive search

!

- Counter-examples useful for bug tracking

- E.g. with rule conflict

!36

Formal analysis approach

!37

Counterexamples

PAT Model Checker

Verification: Safety,

Liveness

Modelling: Environment,

System Design

Stakeholders: Nurses,

Engineers

Collecting Patient behaviours

Collecting Requirements

Labelled transition systems- Modelling of people’s behaviours

!38

Labelled transition systems- Modelling of system's behaviours

!39

Properties specification- P1: Deadlock freeness

- P2: Guaranteed reminders

!

- P3: System inconsistency

!

!

- P4: Conflicted or erroneous reminders

!40

#assert SmartBedroom() |= [](UsingWrongBed -> <>RemindedWrongBed);

#define InConsistency (Pos_Person[1] == SHOWROOM

&& ShowerFlag && sensors[PIR] == SILENT);

# define ConflictReminder (ReminderStage[ShowerNoSoap * 2] != 0

&& ReminderStage[WanderingInSR * 2] != 0);

Verification result- Run on Intel Xeon CPU E5506 2.13GHz, 32GB RAM

- Total runtime around 3.5hours

!41

Bugs? Property Result Nb of states Nb of transitions Execution time (s)

- P1.1 - - - OOM

J P1.2 True 1,433,654 2,038,064 815

J P1.3 True 10,783,353 15,832,370 7,045

J P2.1 True 1,599,797 2,430,351 1,945

L P2.2 False 68,178 130,734 39

L P2.3 False 2,192,251 4,531,005 12,414

L P2.4 False 832,144 1,663,779 729

L P2.5 False 4,314 5,150 1.6

J P2.6 True 1,579,579 2,377,381 1,913

L P3 True 572 745 0.3

L P4.1 True 14,675 20,615 6.1

L P4.2 True 2,446 3,036 1.11

L P4.3 True 2,332,744 3,001,756 1,047

Verification result

!42

System part # Rules # Non-reachable # Redundant/ #Duplicated # Conflict

Bedroom 17 2 8/3 2

Shower Room 22 5 16/2 -

Avg. Time (s) - 2.05 3.05 -

- System split in two for memory purpose

Bug report with error trace- System inconsistency

- enterShowerRoom.1 -> turnOnTap -> exitShowerRoom.1 -> port.PIRShowerRoom.Silent

!

- False alarm

- enterBedroom.2 -> sitOnBed.2.1 -> promptReminder.1

!43

OutlineI. Introduction

II. Context reasoning

- Reasoning strategies

- Reasoning architecture

III. Validation & Verification

IV.Conclusion

!44

Conclusion- Contribution of model checking to AAL

- Formal verification and certification of system

- Useful bug tracking tool at design level

- Save time on costly deployments

- Avoid patient disturbance of early testing

!

- An excellent verification tool during system design

- To be used in conjunction with real deployments

!45