Devising Your Data Movement Strategy for IoT
-
Upload
solace -
Category
Technology
-
view
630 -
download
0
Transcript of Devising Your Data Movement Strategy for IoT
![Page 1: Devising Your Data Movement Strategy for IoT](https://reader035.fdocuments.net/reader035/viewer/2022062523/58708c821a28ab412b8b48db/html5/thumbnails/1.jpg)
1Copyright SolaceConfidential
Devising your Data Movement
Strategy for IoTDave McAllister
September, 2016
![Page 2: Devising Your Data Movement Strategy for IoT](https://reader035.fdocuments.net/reader035/viewer/2022062523/58708c821a28ab412b8b48db/html5/thumbnails/2.jpg)
2Copyright SolaceConfidential
Data Movement Between Elements of IoT
Top level system considerations
Architectures for different types of data movement
Which standards to use;where and why
![Page 3: Devising Your Data Movement Strategy for IoT](https://reader035.fdocuments.net/reader035/viewer/2022062523/58708c821a28ab412b8b48db/html5/thumbnails/3.jpg)
3Copyright SolaceConfidential
Relevant System Requirements
Scale of Devices
Capabilities ofDevices
Message Exchange Patterns
Messaging QoS
TrafficCharacteristics
SecurityIT Application Architecture & Technologies
![Page 4: Devising Your Data Movement Strategy for IoT](https://reader035.fdocuments.net/reader035/viewer/2022062523/58708c821a28ab412b8b48db/html5/thumbnails/4.jpg)
4Copyright SolaceConfidential
ProblemStatement
Firewall
Firewall
LoadBalancer
? ? ? ?
![Page 5: Devising Your Data Movement Strategy for IoT](https://reader035.fdocuments.net/reader035/viewer/2022062523/58708c821a28ab412b8b48db/html5/thumbnails/5.jpg)
5Copyright SolaceConfidential
![Page 6: Devising Your Data Movement Strategy for IoT](https://reader035.fdocuments.net/reader035/viewer/2022062523/58708c821a28ab412b8b48db/html5/thumbnails/6.jpg)
6Copyright SolaceConfidential
![Page 7: Devising Your Data Movement Strategy for IoT](https://reader035.fdocuments.net/reader035/viewer/2022062523/58708c821a28ab412b8b48db/html5/thumbnails/7.jpg)
7Copyright SolaceConfidential
Message Bus
![Page 8: Devising Your Data Movement Strategy for IoT](https://reader035.fdocuments.net/reader035/viewer/2022062523/58708c821a28ab412b8b48db/html5/thumbnails/8.jpg)
8Copyright SolaceConfidential
IoT Interlude
http://imgs.xkcd.com/comics/driving.png
![Page 9: Devising Your Data Movement Strategy for IoT](https://reader035.fdocuments.net/reader035/viewer/2022062523/58708c821a28ab412b8b48db/html5/thumbnails/9.jpg)
9Copyright SolaceConfidential
Firewall
Connection Tier
LoadBalancer
EDGE
FOG
Typical IoT CloudDeploymentArchitecture
Enterprise BusFirewall
CORE
![Page 10: Devising Your Data Movement Strategy for IoT](https://reader035.fdocuments.net/reader035/viewer/2022062523/58708c821a28ab412b8b48db/html5/thumbnails/10.jpg)
10Copyright SolaceConfidential
Firewall
Connection Tier
LoadBalancer
EDGE
FOG
System Considerations• Connections, queues,
subscription• MEPs
• Device initiated pub/sub
• Device initiated request/reply
• Cloud app initiated pub/sub
• Cloud app initiated request/reply
• QoS – MQTT QoS0, QoS1• Single point of contact for
apps
Enterprise BusFirewall
CORE
![Page 11: Devising Your Data Movement Strategy for IoT](https://reader035.fdocuments.net/reader035/viewer/2022062523/58708c821a28ab412b8b48db/html5/thumbnails/11.jpg)
11Copyright SolaceConfidential
Enterprise Bus
Connection Tier
System Considerations
Load Balancer
o Connections, queues, subscrptns
o MEPs‐ Device-initiated pub/sub‐ Device-initiated
request/reply‐ Cloud app-initiated pub/sub‐ Cloud app-initiated
request/replyo QoS – MQTT QoS0, QoS1o Single point of contact for
apps
![Page 12: Devising Your Data Movement Strategy for IoT](https://reader035.fdocuments.net/reader035/viewer/2022062523/58708c821a28ab412b8b48db/html5/thumbnails/12.jpg)
12Copyright SolaceConfidential
Enterprise Bus
Connection Tier
Load Balancer
System Considerationso Connections, queues,
topicso MEPs
‐ Device-initiated pub/sub‐ Device-initiated
request/reply‐ Cloud app-initiated pub/sub‐ Cloud app-initiated
request/replyo QoS – MQTT QoS0, QoS1o Single point of contact for
apps
![Page 13: Devising Your Data Movement Strategy for IoT](https://reader035.fdocuments.net/reader035/viewer/2022062523/58708c821a28ab412b8b48db/html5/thumbnails/13.jpg)
13Copyright SolaceConfidential
Enterprise Bus
Connection Tier
Load Balancer
Challenges with Stateless Connect Layero Device Queues
‐ Where is the queue for each device for QoS1 subscriptions?
‐ How does Connection layer find its queue/state? (among the millions….)
o App to device MEP‐ How does Core layer find
Connection layer server for given MQTT device for QoS0?
‐ Where is the Q for a given device for QoS1?
‐ Need massive subscription scalability updated in real time
![Page 14: Devising Your Data Movement Strategy for IoT](https://reader035.fdocuments.net/reader035/viewer/2022062523/58708c821a28ab412b8b48db/html5/thumbnails/14.jpg)
15Copyright SolaceConfidential
MQTT Paho
Proprietary Proprietary/JMS/JCA
HTTP/REST In Runtime
AMQP 1.0 Qpid / JMS
CoAP Several OSS
P R O T O C O L A P IRelevantProtocols& APIs
![Page 15: Devising Your Data Movement Strategy for IoT](https://reader035.fdocuments.net/reader035/viewer/2022062523/58708c821a28ab412b8b48db/html5/thumbnails/15.jpg)
16Copyright SolaceConfidential
MQTT versus CoAPMQTT CoAP
Many to many (but Pub-sub) One to one
Event based State based
No discovery (Pub-sub) Supports content negotiation
OASIS / ISO IETF
3 levels QoS 2 levels QoS
![Page 16: Devising Your Data Movement Strategy for IoT](https://reader035.fdocuments.net/reader035/viewer/2022062523/58708c821a28ab412b8b48db/html5/thumbnails/16.jpg)
17confidential
o Device Constraintso Message Exchange
Patternso Messaging QoSo Traffic Characteristicso Security
Fog
![Page 17: Devising Your Data Movement Strategy for IoT](https://reader035.fdocuments.net/reader035/viewer/2022062523/58708c821a28ab412b8b48db/html5/thumbnails/17.jpg)
18confidential
Fog
MQTT Paho
Proprietary Proprietary/JMS/JCA
HTTP/REST In Runtime
AMQP 1.0 Qpid / JMS
CoAP Several OSS
P R O T O C O L A P I
![Page 18: Devising Your Data Movement Strategy for IoT](https://reader035.fdocuments.net/reader035/viewer/2022062523/58708c821a28ab412b8b48db/html5/thumbnails/18.jpg)
19Copyright SolaceConfidential
Edgeo # Concurrent
Connectionso “Always on”
Connections?o Burstinesso Congestion /
Prioritizationo Multi-protocol?o QoSo Security
![Page 19: Devising Your Data Movement Strategy for IoT](https://reader035.fdocuments.net/reader035/viewer/2022062523/58708c821a28ab412b8b48db/html5/thumbnails/19.jpg)
20Copyright SolaceConfidential
MQTT Paho
Proprietary Proprietary/JMS/JCA
HTTP/REST In Runtime
AMQP 1.0 Qpid / JMS
CoAP Several OSS
P R O T O C O L A P I
Edge
![Page 20: Devising Your Data Movement Strategy for IoT](https://reader035.fdocuments.net/reader035/viewer/2022062523/58708c821a28ab412b8b48db/html5/thumbnails/20.jpg)
21Copyright SolaceConfidential
o Less scale, more concentration
o Need enterprise features
o Enterprise grade, popular APIs
o Support streaming & microservices
Core
![Page 21: Devising Your Data Movement Strategy for IoT](https://reader035.fdocuments.net/reader035/viewer/2022062523/58708c821a28ab412b8b48db/html5/thumbnails/21.jpg)
22Copyright SolaceConfidential
Core
MQTT Paho
Proprietary Proprietary/JMS/JCA
HTTP/REST In Runtime
AMQP 1.0 Qpid / JMS
CoAP Several OSS
P R O T O C O L A P I
![Page 22: Devising Your Data Movement Strategy for IoT](https://reader035.fdocuments.net/reader035/viewer/2022062523/58708c821a28ab412b8b48db/html5/thumbnails/22.jpg)
23Copyright SolaceConfidential
IoT is all about the data
Capacity &Availability Limits
Big Data RIVERShock Absorber
NetworkProcessingStorage
OutagesUpgrades
InconsistentAggregate
InputStream
![Page 23: Devising Your Data Movement Strategy for IoT](https://reader035.fdocuments.net/reader035/viewer/2022062523/58708c821a28ab412b8b48db/html5/thumbnails/23.jpg)
24Copyright SolaceConfidential
PublicClouds
PublicCloudAp
pApp
App
Big Data River
Big Data
Private Cloud
App App App
App App App
PaaS
IaaS
App App App
PublicCloud
On Premise
![Page 24: Devising Your Data Movement Strategy for IoT](https://reader035.fdocuments.net/reader035/viewer/2022062523/58708c821a28ab412b8b48db/html5/thumbnails/24.jpg)
25Copyright SolaceConfidential
Enterprise Bus
Connection Tier
Addressing Scalabilityo Command & Control
sending message to a specific device
![Page 25: Devising Your Data Movement Strategy for IoT](https://reader035.fdocuments.net/reader035/viewer/2022062523/58708c821a28ab412b8b48db/html5/thumbnails/25.jpg)
26Copyright SolaceConfidential
Enterprise Bus
Connection Tier
Addressing Scalabilityo Command & Control
sending message to a specific device
o Device-initiated request/reply
o Among millions of deviceso Across 2 middleware tierso Need Integrated
addressing between Enterprise Bus & Connection Tier at massive scale
![Page 26: Devising Your Data Movement Strategy for IoT](https://reader035.fdocuments.net/reader035/viewer/2022062523/58708c821a28ab412b8b48db/html5/thumbnails/26.jpg)
27Copyright SolaceConfidential
Enterprise Bus
Connection Tier
Addressing Scalability
Load Balancer
![Page 27: Devising Your Data Movement Strategy for IoT](https://reader035.fdocuments.net/reader035/viewer/2022062523/58708c821a28ab412b8b48db/html5/thumbnails/27.jpg)
28Copyright SolaceConfidential
Enterprise Bus
Connection Tier
Addressing Scalabilityo Connection Tier
Addressing‐ MQTT Topics
o Enterprise Bus Addressing‐ Other topics & queues
o Need to scale ‐ In an integrated manner‐ Update in real-time‐ Handling bursts of changes
![Page 28: Devising Your Data Movement Strategy for IoT](https://reader035.fdocuments.net/reader035/viewer/2022062523/58708c821a28ab412b8b48db/html5/thumbnails/28.jpg)
29Copyright SolaceConfidential
Singapore Land Transport Authority – ERP2http://www.straitstimes.com/singapore/transport/ncs-mhi-to-build-islandwide-satellite-based-erp-for-556m
Routing Server
Payments
Analytics
0 0 0
• 200K Concurrent Connections
• MQTT Termination
• Request/reply forfirmware upgrades
• Push Notifications
• Persistent Messaging
• Shock Absorber
• High Throughput
• Fanout
• Slow Consumer Handling
MQTT JMS
MHI OBU
Next-Gen Electronic Road Pricingo Connect 1.5M vehicles
(200K at a time) in real-time for pay per use road tolls
o Suggest better routes, traffic management – real time and long term and value-added services/analytics
![Page 29: Devising Your Data Movement Strategy for IoT](https://reader035.fdocuments.net/reader035/viewer/2022062523/58708c821a28ab412b8b48db/html5/thumbnails/29.jpg)
30Copyright SolaceConfidential
Bus Monitoringo Collect real-time
location, loading information and more thousands of buses
o Optimize bus movement, fleet management, avoid bunching, forecast arrival time and more
REST/MQTT
Regulator’s Dashboard
Commuter’s Apps and
RIAs
Analytics
DWH
ESB (if needed)
CEP
JMS
Legacy REST
Legacy Gateway
REST/MQTT
![Page 30: Devising Your Data Movement Strategy for IoT](https://reader035.fdocuments.net/reader035/viewer/2022062523/58708c821a28ab412b8b48db/html5/thumbnails/30.jpg)
31Copyright SolaceConfidential
Overall Architecture Requirementso Scalability: connections,
queues, subscriptionso “single bus” connectivity for
both apps & devices – solve the whole problem!‐ Unified addressing between
cloud apps & devices‐ Single point of application
connectivityo Support necessary QoS
o Support all MEPs(?)‐ Device-initiated pub/sub‐ Device-initiated request/reply‐ Cloud app-initiated pub/sub‐ Cloud app-initiated
request/reply
IoT Connectivity Concerns
![Page 31: Devising Your Data Movement Strategy for IoT](https://reader035.fdocuments.net/reader035/viewer/2022062523/58708c821a28ab412b8b48db/html5/thumbnails/31.jpg)
32Copyright SolaceConfidential
At the Edgeo Scalability: connections,
queues, subscriptionso Protocols: MQTT, REST,
CoAP(?)o Security:
‐ Authentication & Encryption‐ Per-device authorization
‐ deviceID in topics‐ /in/<devID>/<system> ACL on
/in/<devID>/>
In the Coreo Single, simple point of
contact to talk to any deviceo APIs & protocols to cloud
apps‐ JMS, AMQP, Node.js, REST‐ App-to-device and app-to-app
o Message filteringo Application horizontal
scaling with non-exclusive queues
o Integration with Hadoop
IoT Connectivity Concerns(cont)
![Page 32: Devising Your Data Movement Strategy for IoT](https://reader035.fdocuments.net/reader035/viewer/2022062523/58708c821a28ab412b8b48db/html5/thumbnails/32.jpg)
33Copyright SolaceConfidential
Can’t predict coming evolution of IoT,and changing infrastructure isn’t cheap.
Need flexible data movement platform
![Page 33: Devising Your Data Movement Strategy for IoT](https://reader035.fdocuments.net/reader035/viewer/2022062523/58708c821a28ab412b8b48db/html5/thumbnails/33.jpg)
34Copyright SolaceConfidential
Strong Architecture• Decoupling of concerns
• Scales to meetneeds of clients
• Right featuresat each layer
Smart Use of Standards• Right standards• Features & exchange patterns
• Optimized for needs of users
• Security• Supported by key products andopen source projects
![Page 34: Devising Your Data Movement Strategy for IoT](https://reader035.fdocuments.net/reader035/viewer/2022062523/58708c821a28ab412b8b48db/html5/thumbnails/34.jpg)
35Copyright SolaceConfidential
Thanks!
Dwmcallister [email protected]
Download the free VMR Community Edition at
http://dev.solace.com