Semantic Streams: a Framework for Composable Semantic Interpretation of Sensor Data Kamin Whitehouse...

Semantic Streams: a Framework for Composable Semantic

Interpretation of Sensor Data

Kamin WhitehouseUC Berkeley

EWSN, Feb 13, 2006

Joint with Feng Zhao and Jie Liu, Microsoft Research

fact

DataProcessing

DataProcessing

fact

fact

DataProcessing

Motivation

• Imaging that sensors cover the globe

• Can we automatically query for world facts?

• No: each fact is an interpretation of data

• Processing must consider– Sensor context– Sensor fusion– Data formats– Calibration– Signal transforms– Etc.

World Sensor

DB

fact

DataProcessing

fact

DataProcessing

Semantic Streams• Goal: to allow direct

querying of facts• Example query: “Is a

vehicle in the parking lot”• Instantiates a graph of

inference units to derive the desired fact

• Output is a stream of all vehicles detected

Fact: vehicle

SizeInference

VehicleInference

SpaceInference

Fact: empty space

Sensor.data=72Sensor.data=56Sensor.data=98obj.size=23 obj.size=23Sensor.data=49obj.type=vehicle

Usage Model

• Fixed Sensor infrastructure– Many simultaneous, unrelated users – Short-term queries repeatedly over long periods– Queries are often similar (though not identical)

• When programmer poses new query, system will use existing sensors and inference units, if possible.

• Otherwise, system may give actionable error messages, – need to add 2 more sensors in area XYZ– need to add new inference unit to the system

• Semantic values produced by infrastructure grows organically as it is used for new purposes

Complete System

MobileApplication

QueryQuery

ProcessorInference Graph

Embedding Engine

MSTMLExecution

Engine

Sensors

Planning time

User

OutputStream

Inference UnitInstances

Run time

Background: Prolog and FOL

• george, jane, carA• X, Y, Z, StudentA• female(jane), parent(X,Y)• parent(george,jane).

male(george).• father(X) :- parent(X,Y),male(X).• male(X).

> X = george• father(X).

> X = george• father(jane).

> false

• Constants• Variables• Predicates• Facts• Rules• Queries• Query processing

Markup Languagesensor(magnetometer,

[[60,0,0][70,10,10]]).

sensor(camera,

[[40,0,0][55,15,15]]).

sensor(breakBeam,

[[10,0,0][12,10,2]]).

inference( speedDetector,

needs(

sensor(magnetometer, R) ),

creates(

stream(X),

isa(X,object),

property(X,T,time),

property(X,R,region),

property(X,S,speed) ) ).

• Sensors are declared as logical facts– All regions are appx’d by

3D cubes

• Inference units are logical rules– Antecedents: the fact

streams that it needs– Consequents: the fact

streams that it creates– Spatial relationship

encoded in variable name

Stream Relationships

subregion( A, B )

inference(speedDetector,

needs(

sensor(magnetometer, R) ),

creates(

stream(X),

isa(X,object),

property(X,T,time),

property(X,R2,region),

subregion(R2,R),


• More sophisticated spatial relationships– Subregions

Sensor coverage (R)

Queried area (R2)

Stream Relationshipsinference( speedDetector,

needs(

sensor(breakBeam, R1),



subregion(R1,R),

subregion(R2,R),

subregion(R3,R),

\+ intersecttion(_,R1,R2),

\+ intersecttion(_,R1,R3),

\+ intersecttion(_,R2,R3) ),

creates(

stream(X),

isa(X,object),

property(X,T,time),

property(X,R,region) ) ).

• More sophisticated spatial relationships– Subregions– Non-intersection

Queried area

Sensor coverage

Stream Relationships

inference( vehicleDetector,

needs(

sensor(magnetometer, R),

stream(X),

isa(X,object),

property(X,S,speed) ),

creates(

stream(X),

isa(X,vehicle) ) ).

• More sophisticated spatial relationships– Subregions– Non-intersection

• Stream identity

Query processing

1. Choose first predicate in query

2. Search KB for sensors that match

3. Search output of inference units for streams that match

4. Add new input streams to query

5. Back to step 1

Region(A,[[10…]])isa(A,vehicle)

Stream(X),Isa(X,vehicle)

stream(X),property(X,speed)

Sensor(mag,…)

VehicleInference

stream(Y),property(Y,speed)

Sensor1 Sensor2 Sensor3

SpeedInference

Query processing

• Actionable Error Messages– When missing

inference unit– When missing

sensor


Stream(X),Isa(vehicle,X)


Sensor(mag,…)



VehicleInference

SpeedInference

Query processing

• When too many choices– Multiple graphs

provide logically equivalent results




Sensor(mag,…)



VehicleInference

SpeedInference


Camera

SpeedInference

Quality of Service Constraints

subregion( A, B )


needs(

sensor(camera, R) ),

creates(

stream(X),

isa(X,object),

property(X,T,time),


property(X,S,speed),

property(X,C,confidence),

{C > 99},

property(X,L,latency),

{L < 450} ) ) .

• Inference unit may specify QoS of output– Confidence– Latency

Quality of Service Constraintssubregion( A, B )


needs(

sensor(breakBeam, R1) ),



creates(

stream(X),

isa(X,object),

property(X,T,time),



property(X,C,confidence),

{C > 80},

property(X,L,latency),

{L < 75} ) ) .


Quality of Service Constraints

subregion( A, B )

inference(vehicleDetector,

needs(

sensor(magnetometer, R1),

stream(X),

property(X,S,speed),

property(X,C2,confidence),

property(X,L2,latency),

creates(

stream(X),

isa(X,vehicle),

{C > 0.9 * C2},

{L = L2 + 125} ) ).


• May be function of input QoS

Query processing

• Query can declare QoS parameters to create ordering of inference graphs




Sensor(mag,…)



VehicleInference

SizeInference


Camera

SizeInference

{C>70, L <300}

Example Infrastructure

Goal Application: Vehicle Detection

Query and Inference Graphisa(Y,histogram),property(Y,T,value),property(X,T,time),

isa(X,vehicle),region(X,R,[[10,10,0][40,30,12]]),Isa(Z,photo),property(Z,X,triggerStream)

VehicleInference

SpeedInference

DirectionInference

SizeInference

HistogramService

PhotoService

Break Beams Magnetometers Camera

Screenshot

Final Application

Camera image

Report (histogram)

Break beam plot

Micro Server Camera Server

Magnetometer plot

Conclusion

• Semantic Streams– Infers facts from data– Intended for sensor infrastructure– Can select inference graph based on QoS constraints

• Limitations– Must be a large supply of inference units– Stream operators are difficult to program

• Different programming language/execution model• Programmer must deal with temporal values, implement

appropriate buffering, etc.

Future Work: Streaming KB’s

Relational Database : Knowledge Base ::

Streaming Database : Streaming KB

SQL operators: Select, Join, MaxOperate on Tables

Logical rules: c(X) :- a(x),b(x).Operate on Facts

Infinite tables Infinite facts

Video: Vehicle Detection

Video: Human Filtering

Traffic Histogram

Semantic Streams: a Framework for Composable Semantic Interpretation of Sensor Data Kamin Whitehouse...

Documents

Transcript of Semantic Streams: a Framework for Composable Semantic Interpretation of Sensor Data Kamin Whitehouse...