Rule-based Real-Time Activity Recognition in a Smart Home Environment

Przemyslaw Woznowski Grigoris Antoniou

10th International Web Rule Symposium (RuleML) 2016, Stony Brook, New York, USA

George Baryannis

Outline

2

Introduction

• Activity Recognition and the Internet of Things• The SPHERE Project• Related Work

Rule-based ADL Recognition

• Offline Version• Online Version• Experimental Evaluation

Conclusions & Future Work

Activity Recognition and the Internet of Things• Sensors have become cheaper, small, widely available• Interconnected within an Internet of Things (IoT)

setting, benefitting from– Distribution of resources– Support for common naming schemas and ontologies– Common access strategies– Availability of computational resources

• Automated Activity Recognition (AR) requires a fusion of multiple sensor-related low-level events

• Challenge: to locate and fuse the right pieces of information from an IoT instance (e.g. sensor network) in order to realise AR at the best quality of information possible

3

Approaches for sensor-based AR

• Data-driven – exploiting machine learning techniquesNoise and uncertainty are handled wellRequire large, annotated training datasetsData conflicts are not handled well

• Knowledge-driven – leveraging logical modelling and reasoning

No training data neededNot as robust against noise and uncertaintyRequire carefully crafted rules

4

Activity Recognition in a multi-modalsmart home environment• Focuses on the so-called Activities of Daily Living

(ADL), with the purpose of supporting Ambient Assisting Living (AAL) efforts– Long-term monitoring of health-related features– Direct assistance

• Main requirements– Increased need for robustness against noise (due to multiple

sensors)– Support for complex, uncertain and non-sequential scenarios– Support for user localization within the smart home, with

minimal user involvement– Inference of real-time, continuous streams of meaningful and

actionable events– Less reliance on training data, since they are difficult to acquire

due to them being environment-dependent 5

The SPHERE Project

Woznowski et al. (2015)6

• SPHERE: a Sensor Platform for Healthcare in a Residential Environment– Common platform

of non-medical networked sensors

– Deployed on a home environment testbed, the SPHERE house

– Impact a range of healthcare needs simultaneously

• Chen et al.: equivalence and subsumption reasoning on ontologies modelling both sensors and activities

Both offline and real-time modes, incrementally-specific recognitionRequires activities to be performed in a predefined, strictly sequential order and fixed time intervals

• MetaQ: SPARQL-based reasoning on sensor data represented as RDF graphs

Recognition building from atomic gestures to complex activitiesWorks only offline, does not take into account missing activities

Related Work (1)

7

• Skarlatidis et al.: hybrid approach, combining event calculus reasoning with Markov Logic Networks

High recognition rates, robustness against missing dataOnly focuses on posture and movement-related activities, as opposed to complex ADL scenarios

• Helaoui et al.: hybrid approach, employing a probabilistic DL reasoner

Recognition building from atomic gestures to complex activitiesRequires training data, works only offline, no support for temporal features

Related Work (2)

8

Outline

9

Introduction

• Activity Recognition• The SPHERE Project• Related Work

Rule-based ADL Recognition

• Offline Version• Online Version• Experimental Evaluation

Conclusions & Future Work

10

• Rule base– rules defined by examining collected sensor data from scripted experiments

• Fact base– derived from sensor data

• The system operates in two modes– Offline: precollected sensor data are stored as individual facts

• Can provide activity reports for past periods (e.g. hourly or daily)– Real-time: facts represent each deployed sensor node and store

its current state/value (as well as its previous one)• Recognises activities as soon as the associated sensor events

happen

RuleBase

Fact Base

Inference Engine (JESS)

“Expert” knowledge Sensor data

Rule-based System Overview

11

• Environmental Sensors– Door contact, electricity meters, water flow meters, PIR– Ambient light useful only when the effect of sunlight is minimal

(i.e. the sun is below the horizon)– Scripted experiment data do not yield patterns from ambient

noise, dust, humidity and temperature

• Video Sensors– 2D bounding box coordinates– Depth coordinates of 3D bounding box

Fact Base

12

PIR-basedLocation

Camera-based

Location

FusedLocation

Second Level Rules

Higher Level Rules

DoorInteraction

ElectricalDevices

WaterFlow

WaterFlow

Clean-upAtomic Activity Rules

Rule Base

rules assume single inhabitant scenarios

13

• Detect changes in sensor values within their reporting windows– From >0 to 0: OpenDoor / SwitchOff– From 0 to >0: CloseDoor / SwitchOn

Doors and Electrical Devices

14

• Water meters do not have a reporting period, only report instantaneously– Positive flow value: OpenTap– Zero flow value: CloseTap

• “Clean up” rules follow to keep only the earliest events for each distinct opening or closing occurrence– If there is no close tap activity between two consecutive open

tap activities, remove the latest one

Water Flow

15

• Rules so far recognise atomic activities• Higher-level rules progressively combine recognised

activities to infer activities of increasingly higher complexity– SwitchOn(device,t1) and SwitchOff(device,t2) Use(device, t1,

t2)– SwitchOn(tv,t1) and SwitchOff(tv,t2) WatchingTV(t1, t2)– Use of taps in kitchen or bathroom WashHands or WashFace– Use of taps in bathroom BrushTeeth or Bathing/Showering– Use kettle and close tap in kitchen PreparingDrink– Open fridge and use toaster PreparingSnack– PreparingDrink or PreparingSnack and use of taps in kitchen

WashDishes

Complex Activities

16

• Basic PIR rule places user in a specific room, from the time PIR is activated till it’s deactivated– Sequences in the same room merged if temporally close or user

not in a different room in between• Basic video rule places user in a specific room, for as

long as the associated camera reports bounding box coordinates

• Detect ghost sequences since they severely compromise validity– Length of less than 30 frames– Stuck in the same coordinates for more than 30 frames– Width and/or height of box consistently and unjustifiably small,

in correlation with depth

Localisation Rules

17

• Assign confidence values to PIR and camera location reports– PIR: confidence inversely proportional to the number of PIR sensors

simultaneously reporting motion– Video: confidence depends on the probability of being a ghost, based on

the detection heuristics• If only a single source reports a location, it is assumed to hold

(with the associated confidence value)• If PIR and video report the same location, it is assumed to hold

(with confidence values summed)• If PIR and video disagree, the correct location is the one

associated with a recognised atomic activity• If both or neither disagreeing reports are supported by an

activity, we assume the one with the higher confidence holds– If confidences are equal, we trust PIR

Fused Location

18

• Facts now represent the state of each distinct sensor– Instead of the history of sensor events– To detect state change, previous state is also stored

• Changes are necessary only for rules at the lowest level– Second and higher-level rules remain unchanged

• Transparent to the way sensor events are generated• Any state change event is linked to a related atomic

activity– Holds for DC sensors, electricity and water flow meters– Rules fire only once when sensor values change – no need for

“clean up” rules

From Offline to Real-time

19

• Each consecutive activation/deactivation of a PIR sensor corresponds to the user being in the associated room

• Subsequent activations extend the user’s stay when– Activation directly follows the last deactivation– The elapsed time between them does not exceed a threshold– No activation has taken place in a different room in the

meantime• State-based approach is not applicable for video sensors

– Video cameras do not broadcast a single value• Each reported bounding box is stored briefly

– Combined to create facts that represent a period of stay in a room

– The same heuristics used for ghost detection

Online Localisation (1)

20

• Each time a PIR sensor is activated, the system fuses available information to decide on its validity– If there is no active video sequence and no activity detected, we

assume PIR is valid– If the active video sequence with the highest confidence agrees

with PIR, we conclude the user is in the room, summing confidence values

– If video reports a different room, we assume the user is in the room where the most recently recognised atomic activity was performed

Online Localisation (2): Fusion

21

• Offline and real-time versions implemented in Java, using Jess as a rule engine– Implemented rules designed to accommodate variable reporting

periods• Real-time version built as an MQTT client

– Sensor messages are broadcast in separate threads• 10 participants executed an ADL script of half-hour

duration, twice, in the SPHERE house– Ground-truth data acquired through annotation of video images

collected using a head-mounted camera– Subset of performed activities that are recognised: interaction

with doors, electrical devices and water taps, preparing a snack/drink, washing hands/dishes, brushing teeth, bathing/showering

Implementation and Data Collection

22

• TP (true positive): activity performed and recognised• FP (false positive): activity not performed but recognised• FN (false negative): activity performed but not recognised• ,

Evaluation Results

Outline

23

Introduction

• Activity Recognition• The SPHERE Project• Related Work

Rule-based ADL Recognition

• Offline Version• Online Version• Experimental Evaluation

Conclusions & Future Work

Concluding remarks

• Rule-based system capable of operating on both historical and real-time, multi-modal sensor data acquired in a smart home– Bottom-up, multi-level rules to support complex ADL scenarios– Non-deterministic patterns to account for missing activities

• Sensor fusion and heuristics to achieve robustness against noise– 95% recall and 88% precision on average for a significant subset

of activities– 93% room-level localisation accuracy due to effective ghost

detection and location fusion rules

24

25

• Integrate wearable sensor data– Infer activities unidentifiable with only the other sensors– Improve localisation accuracy or provide an alternative to video

cameras when they are not available/allowed• Explore multi-inhabitant scenarios

– Use localisation results to pin down activities to the person performing them

– For some activities, localisation needs to be more fine-grained than room-level

• Explore hybrid approach with Machine Learning research within SPHERE– Incorporate rules as features in ML algorithms– Use rules that act on the results of ML algorithms– Devise ML techniques to learn rules

Current and Future Work

26

Thank you! Questions?

g.bargiannis@hud.ac.ukhttp://www.irc-sphere.ac.uk

27

• Chen, L., Nugent, C.D., Wang, H.: A knowledge-driven approach to activity recognition in smart homes. IEEE Trans. Knowl. Data Eng. 24(6), 961–974 (2012)

• Filippaki, C., Antoniou, G., Tsamardinos, I.: Using constraint optimization for conflict resolution and detail control in activity recognition. In: Keyson, D.V., Maher, M.L., Streitz, N., Cheok, A., Augusto, J.C., Wichert, R., Englebienne, G., Aghajan, H., Krose, B.J.A. (eds.) AmI 2011. LNCS, vol. 7040, pp. 51–60. Springer, Heidelberg (2011)

• Helaoui, R., Riboni, D., Stuckenschmidt, H.: A probabilistic ontological framework for the recognition of multilevel human activities. In: Mattern, F., Santini, S., Canny, J.F., Langheinrich, M., Rekimoto, J. (eds.) UbiComp 2013, pp. 345–354. ACM (2013)

• Meditskos, G., Dasiopoulou, S., Kompatsiaris, I.: MetaQ: a knowledge-driven framework for context-aware activity recognition combining SPARQL and OWL 2 activity patterns. Pervasive Mob. Comput. 25, 104–124 (2016)

• Skarlatidis, A., Paliouras, G., Artikis, A., Vouros, G.A.: Probabilistic event calculus for event recognition. ACM Trans. Comput. Log. 16(2), 11:1–11:37 (2015)

• Woznowski, P., Fafoutis, X., Song, T., Hannuna, S., Camplani, M., Tao, L., Paiement, A., Mellios, E., Haghighi, M., Zhu, N., et al.: A multi-modal sensor infrastructure for healthcare in a residential environment. In: 2015 IEEE International Conference on Communication Workshop, pp. 271–277. IEEE (2015)

References