On the Bulk Ingestion of IoT Devices from Heterogeneous ... · • IoT brokers are software...
Transcript of On the Bulk Ingestion of IoT Devices from Heterogeneous ... · • IoT brokers are software...
![Page 1: On the Bulk Ingestion of IoT Devices from Heterogeneous ... · • IoT brokers are software components that usually follow a topic‐based publish/subscribe paradigm • For each](https://reader030.fdocuments.net/reader030/viewer/2022040608/5ec65d568fdc5d104a35bb2e/html5/thumbnails/1.jpg)
On the Bulk Ingestion of IoT Devices from Heterogeneous IoT Brokers
S. Bonfitto, F. Hachem, E. Belay, S. Valtolina, M. [email protected]
ETSI IoT Workshop ‐‐ 23 October 2019
![Page 2: On the Bulk Ingestion of IoT Devices from Heterogeneous ... · • IoT brokers are software components that usually follow a topic‐based publish/subscribe paradigm • For each](https://reader030.fdocuments.net/reader030/viewer/2022040608/5ec65d568fdc5d104a35bb2e/html5/thumbnails/2.jpg)
The Snap4City project
scalable Smart aNalytic APplication builder for sentient Citieswww.snap4city.org
• Pre‐Commercial Procurement (PCP) project within the H2020 EU Project SELECT4CITIES (http://www.select4cities.eu/)– Design (3 M, 11P), Realization (6 M, 6 P), Prototyping (9 M, 3P)
• Join proposal with the University of Florence (Prof. Paolo Nesi)
• The project goal is to develop an open, standardized, data‐driven, service‐orientedand user‐centric platform that enables large‐scale co‐creation, testing and validation of urban IoE applications and services
• The resulting platform:– Works on the Cloud and exploits micro‐services
– Takes user needs and privacy into account
– Provides tools and community for co‐creation
– Mix data driven, stream and batch processing
![Page 3: On the Bulk Ingestion of IoT Devices from Heterogeneous ... · • IoT brokers are software components that usually follow a topic‐based publish/subscribe paradigm • For each](https://reader030.fdocuments.net/reader030/viewer/2022040608/5ec65d568fdc5d104a35bb2e/html5/thumbnails/3.jpg)
Snap4City Architecture
Smart City Cloud Infrastructure
Km4C
ity Smart C
ity API
Knowledge Base
ETL Processes, Data Analytic, R; IOT App; etc. Data Processing Tools
Development and Management Tools
ETL Processes
Resource Manager
DataGate/CKAN
Km4City Ontology
Phoenix, Hbase+ indexing
Big Data Storage Knowledge
IoT/IoE Applications
AMMALinkedOpen Graph
ServiceMap Data Flow AnalysisDevDash
Elastic Management of Containers
Mobile and Web Apps
Final Users’ Tools
Dashboards
Social Media
IoT/IoE
Open Data
Personal Data
Industry 4.0
GIS + Map Data
IOT / IOE Apps
IOT Directory
Management
Authentication, Authorization, GDPR, Security Assessment
![Page 4: On the Bulk Ingestion of IoT Devices from Heterogeneous ... · • IoT brokers are software components that usually follow a topic‐based publish/subscribe paradigm • For each](https://reader030.fdocuments.net/reader030/viewer/2022040608/5ec65d568fdc5d104a35bb2e/html5/thumbnails/4.jpg)
Cloud-Management of real-time IoT data through the IoT Directory
IOT DirectoryDashboardsIOT Applications
Applications
IOT brokerIOT broker
IOT broker
![Page 5: On the Bulk Ingestion of IoT Devices from Heterogeneous ... · • IoT brokers are software components that usually follow a topic‐based publish/subscribe paradigm • For each](https://reader030.fdocuments.net/reader030/viewer/2022040608/5ec65d568fdc5d104a35bb2e/html5/thumbnails/5.jpg)
What is the bulk ingestion problem?
IOT DirectoryDashboardsIOT Applications
Applications
IOT brokerIOT brokerIOT brokerIOT broker
IOT brokerIOT brokerIOT broker
![Page 6: On the Bulk Ingestion of IoT Devices from Heterogeneous ... · • IoT brokers are software components that usually follow a topic‐based publish/subscribe paradigm • For each](https://reader030.fdocuments.net/reader030/viewer/2022040608/5ec65d568fdc5d104a35bb2e/html5/thumbnails/6.jpg)
Accessing information about devices and their sensors
• IoT brokers are software components that usually follow a topic‐based publish/subscribe paradigm
• For each kind of IoT broker we can create an API interface that can be exploited for– Identifying the available devices and sensors/actuators– Subscribe to the events that they generate
• In this way it is easy the acquisition of the device information by directly querying the IoT broker
IoT
![Page 7: On the Bulk Ingestion of IoT Devices from Heterogeneous ... · • IoT brokers are software components that usually follow a topic‐based publish/subscribe paradigm • For each](https://reader030.fdocuments.net/reader030/viewer/2022040608/5ec65d568fdc5d104a35bb2e/html5/thumbnails/7.jpg)
Data formats and structures
![Page 8: On the Bulk Ingestion of IoT Devices from Heterogeneous ... · • IoT brokers are software components that usually follow a topic‐based publish/subscribe paradigm • For each](https://reader030.fdocuments.net/reader030/viewer/2022040608/5ec65d568fdc5d104a35bb2e/html5/thumbnails/8.jpg)
Semantic Extraction Rules
• Use of extraction rules for identifying the parts of the values produced by the devices that correspond to– basic properties of sensors/actuators– the kind of sensors/actuators (temperatures, humidity, semaphore)– and, properties of the IoT devices (model, type, position, etc…)
• Definition of Ontological concepts by means of which the extracted attributes can be semantically annotated
• In this way we can create a uniform representation of the events generated by heterogeneous IoT devices and make easy their integration
![Page 9: On the Bulk Ingestion of IoT Devices from Heterogeneous ... · • IoT brokers are software components that usually follow a topic‐based publish/subscribe paradigm • For each](https://reader030.fdocuments.net/reader030/viewer/2022040608/5ec65d568fdc5d104a35bb2e/html5/thumbnails/9.jpg)
Example
device
sensor sensor
position
has has
Is_in
devicetype
deviceId
value
type
value type
latitude longitude
name
![Page 10: On the Bulk Ingestion of IoT Devices from Heterogeneous ... · • IoT brokers are software components that usually follow a topic‐based publish/subscribe paradigm • For each](https://reader030.fdocuments.net/reader030/viewer/2022040608/5ec65d568fdc5d104a35bb2e/html5/thumbnails/10.jpg)
The data acquisition process
QUERY IOT BROKER
APPLY EXTRACTION RULES
CHECKVALIDITY STORAGE
![Page 11: On the Bulk Ingestion of IoT Devices from Heterogeneous ... · • IoT brokers are software components that usually follow a topic‐based publish/subscribe paradigm • For each](https://reader030.fdocuments.net/reader030/viewer/2022040608/5ec65d568fdc5d104a35bb2e/html5/thumbnails/11.jpg)
ms
ms
ms
ms
#sensori #sensori
#sensori
Experimental Analysis
![Page 12: On the Bulk Ingestion of IoT Devices from Heterogeneous ... · • IoT brokers are software components that usually follow a topic‐based publish/subscribe paradigm • For each](https://reader030.fdocuments.net/reader030/viewer/2022040608/5ec65d568fdc5d104a35bb2e/html5/thumbnails/12.jpg)
Conclusion and Future Work
• In this paper we presented the problem of bulk ingestion of IoT devices in the Snap4city platform
• We discussed the use of extraction rules for identifying parts of the values generated by IoT devices that can be mapped to concepts of an Ontology
• We proposed the use of bulk update rules for making IoT device representation compliant with the rules imposed in the Snap4city architecture
• As future work we plan to:– Develop a machine learning approach for the semi‐automatic generation of extraction rules
– Identifying suggestions to the user for easily making IoT devices compliant with our internal representation
![Page 13: On the Bulk Ingestion of IoT Devices from Heterogeneous ... · • IoT brokers are software components that usually follow a topic‐based publish/subscribe paradigm • For each](https://reader030.fdocuments.net/reader030/viewer/2022040608/5ec65d568fdc5d104a35bb2e/html5/thumbnails/13.jpg)
Thank you!