Efficient RDF Interchange (ERI) Format for RDF Data Streams

Efficient RDF Interchange (ERI) Format for RDF Data Streams

Javier D. Fernández, Alejandro Llaves, Oscar Corcho

Ontology Engineering Group (OEG)Universidad Politécnica de Madrid, Spain

Outline

Index

1. Introduction & Motivation2. Background3. Efficient RDF Interchange (ERI) Format

i. Basic Conceptsii. ERI Streamsiii. Practical Deployment

4. Evaluation5. Conclusions and Next steps

2

3

INTRODUCTION - Static data versus RDF data streams

3


Transform LoadExtract

Files

DBMS

Spatial Information

Web APIs

Linked Data discovery

3


My datasetAugust 2014

My d

Febru


Files

DBMS

Spatial Information

Web APIs


3



My d

Febru


Files

DBMS

Spatial Information

Web APIs


“Most semantic tools are focused on this static view”

3


RDF Data Streams are gaining momentum, generated from any type of data stream, and combining real-time and historical data.

©Wilgengebroed on Flickr, Mr3641, ProtoplasmaKid and ISA Internationales Stadtbauatelier in commons wikimedia


My d

Febru

3


3


RDF streams: potentially unbounded sequences of timestamped RDF statements or graphs.

3



user1_observation [t1]weather1_observation [t1] user2_observation [t3]…

3



t

u1 u2 u3 u4

w1 w2 w3

Stream

user1_observation [t1]weather1_observation [t1] user2_observation [t3]…

3

INTRODUCTION - Motivation

Achieve efficient transmission of RDF streams, a necessary step to ensure higher throughput for RDF Stream processors

Stream source

Stream source

Stream source

Stream source

Stream Processor

Engine

Historic Information

C-SPARQL, SPARQLStr

eammorph-streamsCQELS Cloud

Ztreamy…

Stream source

queries

Continuous results

INTRODUCTION – Motivation - Requirements

16

Efficient transmission of RDF streams:• Streamable

• Scalable

• Easy (fast) to process (create and parse)

• Compact

• Parametrizable (several tradeoffs compression/time)

BACKGROUND

17

Plain:Turtle/Trig/JSON-LD

Plain+Compression (e.g. gzip) HDT

Streaming HDT RDSZ

RDF/XML + EXI ERI

Streamable Yes Yes No Yes Yes Yes Yes

Scalable Limited Yes Yes No Yes Yes YesEasy (fast) to create and parse Yes Limited Limited Yes Limited Limited Yes

Compact No Yes Yes Limited Yes Yes YesParametrizable: compression/time No Limited Yes No Limited Limited Yes

Outline

Index




18

EFFICIENT RDF INTERCHANGE (ERI) FORMAT – Basic Concepts

19

• (Assumption) Most RDF streams are well structured• the is well-known by the data provider• the number of in the structure are limited

structurevariations


20


• Efficient RDF Interchange (ERI) Format encodes the information at two levels:

structurevariations


21



• A sliding dictionary of structures: Structural Dictionary

structurevariations


22



• A sliding dictionary of structures: Structural Dictionary• The concrete value for each predicate

structurevariations


23




structurevariations


24




structurevariations


25

t

u1 u2 u3 u4

w1 w2 w3

Stream

“7.7”^^xsd:float “9.4”^^xsd:float

Structural

Dictionary

temperature

Casual user

Anual pass

wind

ID-30

ID-31 ID-32

ID-33

…

weather: TemperatureObservation

rdf:type

weather:AirTemperature

ssn:observedProperty

???

ex:CelsiusValue

… …

…


26

t

u1 u2 u3 u4

w1 w2 w3

Stream


Structural

Dictionary

temperature

Casual user

Anual pass

wind

ID-30

ID-31 ID-32

ID-33

…


rdf:type



???

ex:CelsiusValue

… …

…

molecule


27

t

u1 u2 u3 u4

w1 w2 w3

Stream


Structural

Dictionary

temperature

Casual user

Anual pass

wind

ID-30

ID-31 ID-32

ID-33

…


rdf:type



???

ex:CelsiusValue

… …

…

molecule


28

t

u1 u2 u3 u4

w1 w2 w3

Stream


Structural

Dictionary

temperature

Casual user

Anual pass

wind

ID-30

ID-31 ID-32

ID-33

…


rdf:type



???

ex:CelsiusValue

… …

…

molecule


29

t

u1 u2 u3 u4

w1 w2 w3

Stream


Structural

Dictionary

temperature

Casual user

Anual pass

wind

ID-30

ID-31 ID-32

ID-33

…


rdf:type



???

ex:CelsiusValue

… …

…

molecule


30

t

u1 u2 u3 u4

w1 w2 w3

Stream


Structural

Dictionary

temperature

Casual user

Anual pass

wind

ID-30

ID-31 ID-32

ID-33

…


rdf:type



???

ex:CelsiusValue

… …

…

molecule


31

t

u1 u2 u3 u4

w1 w2 w3

Stream


Structural

Dictionary

temperature

Casual user

Anual pass

wind

ID-30

ID-31 ID-32

ID-33

…


rdf:type



???

ex:CelsiusValue

… …

…

molecule


32

t

u1 u2 u3 u4

w1 w2 w3

Stream


Structural

Dictionary

temperature

Casual user

Anual pass

wind

ID-30

ID-31 ID-32

ID-33

…


rdf:type



???

ex:CelsiusValue

… …

…

molecule


33

• ERI processing model

• Minimal Information Unit is a molecule:

• We initially restrict to subject molecules

sens-obs:Observation_AirTemperature_4UT01_2003_3_31_6_55_00 a weather:TemperatureObservation ; rdfs: label “Air temperature at 6:55:00”, “Verified” ; om-owl:observedProperty weather:_AirTemperature ; om-owl:procedure sens-obs:System_4UT01 ; om-owl:result sens-obs:MeasureData_AirTemperature_4UT01_2003_3_31_6_55_00 ; om-owl:samplingTime sens-obs:Instant_2003_3_31_6_55_00. ex:CelsiusValue “7.7”^^xsd:float

sens-obs:Observation_AirTemperature_4UT01_2003_3_31_7_45_00 a weather:TemperatureObservation ; rdfs: label “Air temperature at 7:45:00”, “Not Verified” ; om-owl:observedProperty weather:_AirTemperature ; om-owl:procedure sens-obs:System_4UT01 ; om-owl:result sens-obs:MeasureData_AirTemperature_4UT01_2003_3_31_7_45_00 ; om-owl:samplingTime sens-obs:Instant_2003_3_31_7_45_00 . ex:CelsiusValue “9.4”^^xsd:float


34

Sub

ject

M

ole

cule

…

Suu

bje

ctM

ole

cule

…




35

Sub

ject

M

ole

cule

…..Structure ID30= a (1, weather:TemperatureObservation) rdfs:label (2) om-wl:observedProperty (1, weather:_AirTemperature ) om-owl:procedure (1,sens-obs:System_4UT01) om-owl:result (1) om-owl:samplingTime (1) ex:CelsiusValue (1) …..

Structural Dictionary

…

Suu

bje

ctM

ole

cule

…




36

Sub

ject

M

ole

cule



…

Suu

bje

ctM

ole

cule

…

Air Temperature Observations of the

Sensor “System_4UT01”






37

Sub

ject

M

ole

cule



…

Suu

bje

ctM

ole

cule

…

Air Temperature Observations of the

Sensor “System_4UT01”

EFFICIENT RDF INTERCHANGE (ERI) FORMAT – ERI Streams

38Based on: Efficient XML Interchange (EXI) format

Block

Molecule

Molecule

Molecule

…

Block

Molecule

Molecule

Molecule

…

Block

Molecule

Molecule

Molecule

……

Multiplex / Demultiplex

Compression/Decompression (per channel)

StreamHeader

Stream Body

METADATA

COMPCHAN.

COMPCHAN.

COMPCHAN.

COMPCHAN.

COMPCHAN.

COMPCHAN.

METADATA

COMPCHAN.

COMPCHAN.

COMPCHAN.

COMPCHAN.

METADATA

COMPCHAN.

COMPCHAN.

COMPCHAN.

COMPCHAN.

COMPCHAN.

COMPCHAN.

ChannelsStructural Channels

Value Channels…

ERI stream

EFFICIENT RDF INTERCHANGE (ERI) FORMAT – ERI Streams

39

ERI follows an encoding procedure similar to that of the Efficient XML Interchange (EXI) format.

Structural channels: They encode the subjects in each block and, for each one, the structural properties of the related triples, using the dynamic dictionary of structures.• Main Terms of molecules: subject of the grouping.• ID-Structures: ID of the structure of each molecule in the block. The ID

points to the entry in the Structural Dictionary.• New Structures: New entries in the Structural Dictionary.

– Value channels: They encode the concrete data values held by each predicate in the block in a compact fashion.• One channel per different predicate in the block.

• Lists explicit values or use IDs pointing to a sliding object dictionary

stru

ctur

eva

riatio

ns

EFFICIENT RDF INTERCHANGE (ERI) FORMAT – Practical Deployment

40

…sens-obs:MeasureData_Air…55_00sens-obs:Instant_2003…55_00sens-obs:MeasureData_Air…45_00sens-obs:Instant_2003…55_00…

ID-Structures …3030…

ID-pred1 weather: TemperatureObservationID-pred2 ID-pred3 weather:_AirTemperatureID-pred4 sensobs: System_4UT01 ID-pred5ID-pred6ID-pred7

[IDs ofStructures]

[Encoded Structures] [Strings]

… om-owl:samplingTime

ex:CelsiusValue…

Structural Channels

ID-pred2

[Object Values][Meta: strings]

…Air temperature at 6:55:00VerifiedAir temperature at 7:45:00Not Verified…

ID-pred5

[Term IDs][Meta: IDs]

New Terms

[Strings]

…101245…

ID-pred6


12…

Pote

ntial

Com

pres

sion

Differential…

Prefix compressionZlib

Snappy…

Main Terms of Molecules

[Strings]

….sens-obs:Observation_AirTemperature...55_00sens-obs:Observation_AirTemperature...45_00….


Snappy…


Snappy…

ZlibSnappy

…

Differential…

Differential…

…10…

[Bits]

New Structure Marker

New Structures New Predicates

ZlibSnappy

…

[Bits]

New Object MarkerID-pred5

…01…


[Bits]

11…

1211111

ID-pred7

[Object Values][Meta: xsd:float]

Differential…

…7.79.4….

ValueChannels

Pote

ntial

Com

pres

sion


41




[IDs ofStructures]



ex:CelsiusValue…

Structural Channels

ID-pred2



ID-pred5


New Terms

[Strings]

…101245…

ID-pred6


12…

Pote

ntial

Com

pres

sion

Differential…


Snappy…


[Strings]



Snappy…


Snappy…

ZlibSnappy

…

Differential…

Differential…

…10…

[Bits]



ZlibSnappy

…

[Bits]


…01…


[Bits]

11…

1211111

ID-pred7


Differential…

…7.79.4….

ValueChannels

Pote

ntial

Com

pres

sion

Explicit list of values


42




[IDs ofStructures]



ex:CelsiusValue…

Structural Channels

ID-pred2



ID-pred5


New Terms

[Strings]

…101245…

ID-pred6


12…

Pote

ntial

Com

pres

sion

Differential…


Snappy…


[Strings]



Snappy…


Snappy…

ZlibSnappy

…

Differential…

Differential…

…10…

[Bits]



ZlibSnappy

…

[Bits]


…01…


[Bits]

11…

1211111

ID-pred7


Differential…

…7.79.4….

ValueChannels

Pote

ntial

Com

pres

sion



43




[IDs ofStructures]



ex:CelsiusValue…

Structural Channels

ID-pred2



ID-pred5


New Terms

[Strings]

…101245…

ID-pred6


12…

Pote

ntial

Com

pres

sion

Differential…


Snappy…


[Strings]



Snappy…


Snappy…

ZlibSnappy

…

Differential…

Differential…

…10…

[Bits]



ZlibSnappy

…

[Bits]


…01…


[Bits]

11…

1211111

ID-pred7


Differential…

…7.79.4….

ValueChannels

Pote

ntial

Com

pres

sion



44




[IDs ofStructures]



ex:CelsiusValue…

Structural Channels

ID-pred2



ID-pred5


New Terms

[Strings]

…101245…

ID-pred6


12…

Pote

ntial

Com

pres

sion

Differential…


Snappy…


[Strings]



Snappy…


Snappy…

ZlibSnappy

…

Differential…

Differential…

…10…

[Bits]



ZlibSnappy

…

[Bits]


…01…


[Bits]

11…

1211111

ID-pred7


Differential…

…7.79.4….

ValueChannels

Pote

ntial

Com

pres

sion



45




[IDs ofStructures]



ex:CelsiusValue…

Structural Channels

ID-pred2



ID-pred5


New Terms

[Strings]

…101245…

ID-pred6


12…

Pote

ntial

Com

pres

sion

Differential…


Snappy…


[Strings]



Snappy…


Snappy…

ZlibSnappy

…

Differential…

Differential…

…10…

[Bits]



ZlibSnappy

…

[Bits]


…01…


[Bits]

11…

1211111

ID-pred7


Differential…

…7.79.4….

ValueChannels

Pote

ntial

Com

pres

sion


IDs pointing to a sliding object

dictionary


46




[IDs ofStructures]



ex:CelsiusValue…

Structural Channels

ID-pred2



ID-pred5


New Terms

[Strings]

…101245…

ID-pred6


12…

Pote

ntial

Com

pres

sion

Differential…


Snappy…


[Strings]



Snappy…


Snappy…

ZlibSnappy

…

Differential…

Differential…

…10…

[Bits]



ZlibSnappy

…

[Bits]


…01…


[Bits]

11…

1211111

ID-pred7


Differential…

…7.79.4….

ValueChannels

Pote

ntial

Com

pres

sion



dictionary


47




[IDs ofStructures]



ex:CelsiusValue…

Structural Channels

ID-pred2



ID-pred5


New Terms

[Strings]

…101245…

ID-pred6


12…

Pote

ntial

Com

pres

sion

Differential…


Snappy…


[Strings]



Snappy…


Snappy…

ZlibSnappy

…

Differential…

Differential…

…10…

[Bits]



ZlibSnappy

…

[Bits]


…01…


[Bits]

11…

1211111

ID-pred7


Differential…

…7.79.4….

ValueChannels

Pote

ntial

Com

pres

sion



dictionary

Extraction of types


48




[IDs ofStructures]



ex:CelsiusValue…

Structural Channels

ID-pred2



ID-pred5


New Terms

[Strings]

…101245…

ID-pred6


12…

Pote

ntial

Com

pres

sion

Differential…


Snappy…


[Strings]



Snappy…


Snappy…

ZlibSnappy

…

Differential…

Differential…

…10…

[Bits]



ZlibSnappy

…

[Bits]


…01…


[Bits]

11…

1211111

ID-pred7


Differential…

…7.79.4….

ValueChannels

Pote

ntial

Com

pres

sion



dictionary

Extraction of types


49




[IDs ofStructures]



ex:CelsiusValue…

Structural Channels

ID-pred2



ID-pred5


New Terms

[Strings]

…101245…

ID-pred6


12…

Pote

ntial

Com

pres

sion

Differential…


Snappy…


[Strings]



Snappy…


Snappy…

ZlibSnappy

…

Differential…

Differential…

…10…

[Bits]



ZlibSnappy

…

[Bits]


…01…


[Bits]

11…

1211111

ID-pred7


Differential…

…7.79.4….

ValueChannels

Pote

ntial

Com

pres

sion



dictionary

Extraction of types

Outline

Index




50

EVALUATION - COMPRESSION

51


52

ERI excels in space for streaming and statistical dataset


53


RDSZ remains comparable to our approach


54


The object dictionary can overload the representation, although it always obtains comparable compression ratios.

RDSZ remains comparable to our approach


55


56

A smaller buffer in ERI-1k slightly affects the efficiency

EVALUATION - PARSING

57


58

ERI always outperforms the RDSZ compression time (3 and 3.8 times on average for ERI-4k and ERI-4k-Nodict, respectively)


59


ERI decompression is commonly slower (1.4 times on average in both ERI configurations), typically due to decompressing several channels.


60


ERI decompression is commonly slower (1.4 times on average in both ERI configurations), typically due to decompressing several channels.

Channels could be grouped (as in EXI)

EVALUATION – CONSUMING SCENARIO

61

In parsing: transmission + decompression


62

ERI-4k and ERI-4k-Nodict outperform the baseline in transmission + decompressionexcept for those datasets with less regularities in the structure or the data values,

In parsing: transmission + decompression


63

In a scenario in which we include the compression time


64

ERI-4k suffers an expected overhead as we are always including the timeto process the information



65

ERI-4k suffers an expected overhead as we are always including the timeto process the information


The time in which the client receives all data in ERI is comparable to the baseline

Results

66

• Compressed, efficient RDF interchange (ERI) format• exploit the RDF data stream regularity of their structure and

data values• Flexible and extensible ERI configurations• Minimize transmission costs in RDF stream processing

• State-of-the-art compression • Remains efficient in performance

• Time overheads are relatively low and can be assumed in many scenarios.

67

Next steps

• Integration within RDF streaming Engines • e.g. morph-streams, CQELS Cloud• 3 purposes:

• scaling to higher input data rates• minimizing the data exchange among processing nodes• serving a small set of operators on the compressed data

• Parallel compression/decompression• preliminary proposal on Storm

• Align the proposal with the results of W3C RSP group regarding streaming modeling and serialization

Efficient RDF Interchange (ERI) Format for RDF Data Streams

Javier D. Fernández, Alejandro Llaves, Oscar CorchoOntology Engineering Group (OEG), Universidad Politécnica de Madrid, Spain

purl.org/net/ro-eri-ISWC14

Electronic edition:

Research object:

Efficient RDF Interchange (ERI) Format for RDF Data Streams

Data & Analytics

Transcript of Efficient RDF Interchange (ERI) Format for RDF Data Streams