IoT Protocols Integration

Angelo  Corsaro,  PhD  Chief  Technology  Officer  

angelo.corsaro@prismtech.com

with

IaaS

analytics

operational systems

information systems

mobile

desktop

web

embeddedfog computing

messaging / data-sharing

Cloud Messaging

Fog

Fog

Storage

IoT Systems are heterogeneous by nature

IoT architectures have to be designed so to facilitate integration

The danger of O(N2) integration should be addressed architecturally

Integration

Stable Core

Define a Stable Core made by a Reference Platform and Data Model.

Integrate always toward the Stable Core

This approach ensure that the integration problem retains O(N) complexity

Stable Core

Stable Core with Vortex

Vortex is a universal data-sharing backbone

Cop

yrig

ht P

rism

Tech

, 201

5

Vortex supports natively a large number of mobile, enterprise, embedded, cloud, web, and analytics computing platforms

Mobile & Web

Embedded

General Purpose

ClouD

Private Cloud

Analytics

Vortex is programming language independent

Programming Langauges

Programming Langauges

import dds import timeif __name__ == '__main__': topic = dds.Topic("SmartMeter", "Meter") dw = dds.Writer(topic) while True: m = readMeter() dw.write(m) time.sleep(0.1)

import dds._import dds.prelude._import dds.config.DefaultEntities._

object SmartMeter { def main(args: Array[String]): Unit = { val topic = Topic[Meter](“SmartMeter”) val dw = DataWriter[Meter](topic) while (true) { val meter = readMeter() dw.write(meter) Thread.sleep(SAMPLING_PERIOD) } }}

Vortex is data-centric and natively supports the definition of canonical data models

Data-Modeling

Data-Modeling

         enum  UtilityKind  {            ELECTRICITY,            GAS,            WATER              };                            struct  Meter  {            string  sn;            UtilityKind  utility;            float  reading;            float  error;              };                                      #pragma  keylist  Meter  sn  

Vortex provide an extremely rich technological base to define the infrastructure of an IoT System

Vortex natively supports the definition of canonical data models

In Summary

Integration with Vortex-Gateway

Vortex integration swiss-army knife

Based on Apache Camel and supporting 200+ connectors to protocols, data stores, analytics, etc.

Supports protocol, format and encoding transformation

Extensible to allow quick integration of proprietary technologies

messaging/Data-Sharing

Data Stores

Putting it all Together

IaaS

Storage

Getting Started with Vortex-Gateway

Cop

yrig

ht P

rism

Tech

, 201

5

Architecture

camel-ospl Component

VortexOpenSplice

camel-ddsi Component

Vortex-Cafe

• Based on DataReader & DataWriter • Typed data

• No native or generated code • No marshalling (raw data buffer)

Camel DDS Processors

• DDSI demarshaller / marshaller • Data transformation • Dynamic Poll Enricher • QoS adaptations

Cop

yrig

ht P

rism

Tech

, 201

5

Components: Provide connectivity to a given technology

Endpoints: Represent a source/destination of messages from/to a components

Exchanges: Encapsulate data and meta coming from and end-point

Routes: Define path from input endpoints to output endpoints

Processors: Allow to perform transformation to the data, its format, etc.

Key Elements

endpoint exchange route

processor

Cop

yrig

ht P

rism

Tech

, 201

5

A component is a factory of Endpoint instances. Over 200 components are supported

Endpoints are specified using URI:

- dds:TopicName:DomainID/TopicType?QoS

- cometd://host:port/channelname

- jms:[topic:]destinationName

- [tcp|udp|vm]:host[:port]

- xmpp://host:port/room

- …

Components and Endpoints

Cop

yrig

ht P

rism

Tech

, 201

5

Sample Route in Scala

val route= new RouteBuilder { ! “ddsi:Foo:1/FooType” to “dds:Bar:2/FooType” !} !

DDSI Endpoint ! Topic Name ! DomainID! TypeName!

val route= new RouteBuilder { ! “dds:Foo:1/FooType” to “dds:Bar:2/FooType” !} !

DDS Endpoint ! Topic Name ! DomainID! TypeName!

Cop

yrig

ht P

rism

Tech

, 201

5

EIP provide pre-specified ways of integrating endpoints

The most common integration patterns are supported, such as, multicast, filtering, correlation, content-routing, load-balancing, etc.

Multicast Ex:

Enterprise Integration Patterns

val shapesRoute = new RouteBuilder { ! “dds:Foo:1/FooType” !

"to (“dds:Bar:2/BarType”, “jms:topic:Foo”) !} !

Recipient List Content Based Router

Message FIlter

A B

Correlation ID

Splitter Aggregator

Data Integration

Cop

yrig

ht P

rism

Tech

, 201

5

Data Integration Patterns are a synthesis of the most commonly used techniques for integrating systems that don’t share the same representation for some given entities

Some common Data Integration Patterns are

- Content Enricher

- Content Filter

- Type Transformation

- Canonical Data Model

Data Integration Patterns

Cop

yrig

ht P

rism

Tech

, 201

5

Enriches a data type by adding additional information required by the target system/subsystem

Example: Compute speed and enrich position

Content Enricher

struct CartesianCoord2D { long x; long y; };

struct Dynamics2D { long x; long y; long dx; long dy;

};System A System B

Content Enricher

Cop

yrig

ht P

rism

Tech

, 201

5Projects a data type by removing information not required by the target system/subsystem

Example: Going from Dynamics to Position

Content Filter

struct CartesianCoord2D { long x; long y; };

struct Dynamics2D { long x; long y; long dx; long dy;

};System A System B

Content Filter

Cop

yrig

ht P

rism

Tech

, 201

5

The Canonical Data Model Pattern is commonly used in DDS systems. In this case, data is always transformed into the representation used by the “Canonical Model”. This approach is very useful as it reduces the integration complexity

Example: Coordinate Systems

Canonical Data Model

Type Transformerstruct CartesianCoord2D { long x; long y; };

struct PolarCoord2D { long rho; float theta; };

System A System B

Canonical Data Model

Cop

yrig

ht P

rism

Tech

, 201

5

When using a DDSI endpoint de-marshaling is lazy. Thus in this example data is never de-marshalled but simply sent-over to the target domain, with the result that domain bridging is very efficient!

Topic/Domain Bridging

VortexDomain 1

VortexDomain 0

Topic“Circle”

Topic“Square”

1 val in = "ddsi:Circle:0/ShapeType"2 val out = "ddsi:Square:1/ShapeType"3 4 val shapesRoute = new RouteBuilder {5 in to out 6 }

Cop

yrig

ht P

rism

Tech

, 201

5

process allows to perform arbitrary modifications to the exchange content

As such it can be used to implement arbitrary type transformations

Type Transformation

VortexDomain 1

VortexDomain 0

Topic“Foo”

Topic“Bar”

1 val shapeRoute0 = new RouteBuilder {2 in unmarshal(cdrData) process ({ 3 e => foo2Bar(e.getIn.getBody(classOf[Foo]))})4 to out5 }

Data Policing

Cop

yrig

ht P

rism

Tech

, 201

5

Content Router 1 // Define endpoints 2 val inEndpoint = "ddsi:"+ inTopic +":"+ inDomain +"/" + shapeType 3 val outEndpoint1 = "ddsi:"+ outTopic1 +":"+ outDomain1 +"/" + shapeType 4 val outEndpoint2 = "ddsi:"+ outTopic2 +":"+ outDomain2 +"/" + shapeType 5 6 val cdrData = new DataFormatDefinition(new CDRDataFormat) 7 val shapesRoute = new RouteBuilder { 8 inEndpoint unmarshal(cdrData) choice { 9 when ( e => {10 val s = e.getIn.getBody(classOf[ShapeType])11 if (s.x > s.y) true else false12 }) to outEndpoint113 14 when (e => {15 val s = e.getIn.getBody(classOf[ShapeType])16 if (s.x < s.y) true else false17 }) to outEndpoint218 19 otherwise { to (outEndpoint1, outEndpoint2) }20 }21 22 }

VortexDomain 2

VortexDomain 0

Topic“Circle”

Topic“Circle”

VortexDomain 1

Topic“Circle”

if (x >= y) if (x <= y)

Cop

yrig

ht P

rism

Tech

, 201

5

Load Balancing

VortexDomain 2

VortexDomain 0

Topic“Circle”

Topic“Circle”

VortexDomain 1

Topic“Circle”

load balance

1 // Define endpoints 2 val inEndpoint = "ddsi:"+ inTopic +":"+ inDomain +"/" + shapeType 3 val outEndpoint1 = "ddsi:"+ outTopic1 +":"+ outDomain1 +"/" + shapeType 4 val outEndpoint2 = "ddsi:"+ outTopic2 +":"+ outDomain2 +"/" + shapeType 5 6 // Define a Route using the Java DSL 7 val shapesRoute = new RouteBuilder { 8 override def configure() = 9 from(inEndpoint).sample(period, TimeUnit.MILLISECONDS) 10 loadBalance() roundRobin() 11 to(outEndpoint1, outEndpoint2)12 }

Impedance Adaptation

Cop

yrig

ht P

rism

Tech

, 201

5

Data Sampling

VortexDomain 1

VortexDomain 0

Topic“Circle”

Topic“Circle”

1 per 500 msec

1 // Define endpoints 2 val inEndpoint = "ddsi:"+ inTopic +":"+ inDomain +"/" + shapeType 3 val outEndpoint = "ddsi:"+ outTopic +":"+ outDomain +"/" + shapeType 4 5 // Define a Route using the Java DSL 6 val shapesRoute = new RouteBuilder { 7 override def configure() = 8 from(inEndpoint) sample(period, TimeUnit.MILLISECONDS) 9 to(outEndpoint)10 }

Technology Integration

Cop

yrig

ht P

rism

Tech

, 201

5

JMS Integration

JMS

VortexDomain

Topic“Circle”

Topic“Circle”

To Json

1 val dds = "ddsi:Circle:0/ShapeType"2 val jms =3 "jms:topic:circle?jmsMessageType=Text&deliveryPersistent=false"4 5 val shapeRoute = new RouteBuider {6 from(dds) unmarshal("cdr") marshal().json() to(jms)7 }

Cop

yrig

ht P

rism

Tech

, 201

5

1 // Define endpoints 2 val inEndpoint = 3 "ddsi:"+ inTopic +":"+ inDomain +"/" + shapeType 4 val outEndpoint = 5 "websocket://"+inTopic.toLowerCase + "?sendToAll=true" 6 7 // Define a Route using the Scala DSL 8 val shapesRoute = new RouteBuilder { 9 override def configure() =10 from(inEndpoint) unmarshal("cdr") marshal() json() to(outEndpoint)11 }

WebSocket Integration

VortexDomain

Topic“Circle”

To JSON

Cop

yrig

ht P

rism

Tech

, 201

4

MQTT Integration

MQTT

VortexDomain

Topic“Circle”

Topic“Circle”

To Json

1 val dds = "ddsi:Circle:0/ShapeType"2 val mqtt =3 "mqtt:topic:circle"4 5 val shapeRoute = new RouteBuider {6 from(dds) unmarshal("cdr") marshal().json() to(mqtt)7 }

Cop

yrig

ht P

rism

Tech

, 201

4

HBase IntegrationVortex

Domain

Topic“Circle”

To Json

1 val dds = "ddsi:Circle:0/ShapeType"2 val hbase =3 "hbase:Circle?mappingStrategyName=body&operation=CamelHBasePut""4 5 val shapeRoute = new RouteBuider {6 from(dds) unmarshal("cdr") marshal().json() to(hbase)7 }

Demo

Vortex provides, probably, the best technology base for defining the stable core of an IoT System

Vortex Gateway makes it trivial to integrate other systems, applications and technologies

Vortex

Cop

yrig

ht P

rism

Tech

, 201

5