Data Management for the Internet of Things - The NebulaStream … · 2021. 3. 20. · Zeuch et al.:...
15
Zeuch et al.: The NebulaStream Platform: Data and Application Management for the Internet of Things, CIDR, 2020 The NebulaStream Platform: Data and Application Management for the Internet of Things Steffen Zeuch, Ankit Chaudhary, Bonaventura Del Monte, Haralampos Gavriilidis, Dimitrios Giouroukis, Philipp M. Grulich, Sebastian Breß, Jonas Traub, Volker Markl
Transcript of Data Management for the Internet of Things - The NebulaStream … · 2021. 3. 20. · Zeuch et al.:...
Zeuch et al.: The NebulaStream Platform: Data and Application Management for the Internet of Things, CIDR, 2020
The NebulaStream Platform: Data and Application Management for the
Internet of ThingsSteffen Zeuch, Ankit Chaudhary, Bonaventura Del Monte, Haralampos Gavriilidis,
Dimitrios Giouroukis, Philipp M. Grulich, Sebastian Breß, Jonas Traub, Volker Markl
Zeuch et al.: The NebulaStream Platform: Data and Application Management for the Internet of Things, CIDR, 2020
Our View the IoT:The Internet of Things (IoT) presents a novel computing architecture for data management:
A geo-distributed, highly dynamic, and heterogeneous environment of massive scale.
2
Zeuch et al.: The NebulaStream Platform: Data and Application Management for the Internet of Things, CIDR, 2020
NebulaStreamAn IoT data management system to process
thousands of queries over millions of sensors.
Zeuch et al.: The NebulaStream Platform: Data and Application Management for the Internet of Things, CIDR, 2020
Three questions that everyone should answer
before creating a new system
(Is there a need for an IoT system?)
4
Zeuch et al.: The NebulaStream Platform: Data and Application Management for the Internet of Things, CIDR, 2020
1) What's the new thing about IoT applications?
Continuous Operation under
Constant Evolution
Highly Dynamic Execution Environment
Heterogeneity, Distribution, and Volume of Data and Compute
5
Zeuch et al.: The NebulaStream Platform: Data and Application Management for the Internet of Things, CIDR, 2020
2) Is there even a market for an IoT system?
6
Unlocking the potential of the Internet of Things (McKinsey Report 2015)
Leading the IoT (Gartner 2017)
Zeuch et al.: The NebulaStream Platform: Data and Application Management for the Internet of Things, CIDR, 2020
3) Can’t we just use existing (cloud-)solutions?
P P
Kafka Cluster
Flink Cluster
P
7
There is no general-purpose, end-to-end data management system for the IoT with similar functionality as Flink or Spark.
Zeuch et al.: The NebulaStream Platform: Data and Application Management for the Internet of Things, CIDR, 2020
The answers to the questions are:
Yes IoT is something new Yes there is a big market/demand
No we cannot just use existing systems
8
Zeuch et al.: The NebulaStream Platform: Data and Application Management for the Internet of Things, CIDR, 2020
What are the new research challenges of IoT?
Zeuch et al.: The NebulaStream Platform: Data and Application Management for the Internet of Things, CIDR, 2020
C2: Maintain an overview of the
topology
Query Submission in the IoT
C1: Come up with any plan
within a reasonable
amount of time
C3: Find a suitable base for decisions
Deploy
C4: Partial re-optimization and
alternative plans
10
https://www.vnews.com/chicken-moving-in-south-royalton-vt-27867370
Global Query Plan
IoT Topology
Execution Plan
Zeuch et al.: The NebulaStream Platform: Data and Application Management for the Internet of Things, CIDR, 2020
Do we really have to chase the chickens centrally?
11
https://www.youtube.com/watch?v=TVF2sQ10vaQ
NebulaStream: A hybrid approach to meet different requirements.
C: Provide guarantees in this environment (e.g.
ACID)
Zeuch et al.: The NebulaStream Platform: Data and Application Management for the Internet of Things, CIDR, 2020
Query execution in a highly dynamic environment
Failures are the common case in IoT.
GlobalLocal Neighborhood
NebulaStream: Autonomous node design:
12
https://homediy.eu/schuetzen-sie-ihre-huehner-vor-raubtieren/
Zeuch et al.: The NebulaStream Platform: Data and Application Management for the Internet of Things, CIDR, 2020
How can we cope with a highly heterogeneous IoT?
NebulaStream: Hardware-tailored code generation and a dynamic execution engine.
13
https://www.hagemann.de/biologie/zoologie/bestimmungstafel-tiere-auf-dem-bauernhof
Zeuch et al.: The NebulaStream Platform: Data and Application Management for the Internet of Things, CIDR, 2020
How to handle “IoT Scale” ?
Key Principles:● Reduce complexity● Reduce data volumes● Use resources efficiently
14
NebulaStream: A system for a unified sensor-fog-cloud environment.
Sensor Layer
Fog Layer
Cloud Layer
Zeuch et al.: The NebulaStream Platform: Data and Application Management for the Internet of Things, CIDR, 2020
Summary
Yes we need a new system
http://www.nebula.stream
15
A general-purpose, end-to-end data
management system for the IoT.
https://pixabay.com/de/vectors/tier-vogel-huhn-hen-lustig-159510/
