Retour d'expérience Large IoT project / BigData : détail du cas réel de HagerServices
Bigdata analytics and our IoT gateway
-
Upload
freek-van-gool -
Category
Technology
-
view
128 -
download
0
Transcript of Bigdata analytics and our IoT gateway
![Page 1: Bigdata analytics and our IoT gateway](https://reader034.fdocuments.net/reader034/viewer/2022042707/58e5765d1a28abbf5d8b4971/html5/thumbnails/1.jpg)
2 May 2023 1
Aemro Amare, Freek van Gool
Bigdata Analytics and our IoT Gateway :Toon
![Page 2: Bigdata analytics and our IoT gateway](https://reader034.fdocuments.net/reader034/viewer/2022042707/58e5765d1a28abbf5d8b4971/html5/thumbnails/2.jpg)
Agenda• A little bit about us• Who is Quby & what is Toon?• Our big data journey• Data collection & services• Architecture & technologies• Demo• Questions
@javafreekNL @twoxey
![Page 3: Bigdata analytics and our IoT gateway](https://reader034.fdocuments.net/reader034/viewer/2022042707/58e5765d1a28abbf5d8b4971/html5/thumbnails/3.jpg)
A little bit about us
Freek Aemro
@javafreekNL @twoxey
![Page 4: Bigdata analytics and our IoT gateway](https://reader034.fdocuments.net/reader034/viewer/2022042707/58e5765d1a28abbf5d8b4971/html5/thumbnails/4.jpg)
First attemptHome control box
@javafreekNL @twoxey
![Page 5: Bigdata analytics and our IoT gateway](https://reader034.fdocuments.net/reader034/viewer/2022042707/58e5765d1a28abbf5d8b4971/html5/thumbnails/5.jpg)
PivotEnergy stick
![Page 6: Bigdata analytics and our IoT gateway](https://reader034.fdocuments.net/reader034/viewer/2022042707/58e5765d1a28abbf5d8b4971/html5/thumbnails/6.jpg)
What is Toon?
![Page 7: Bigdata analytics and our IoT gateway](https://reader034.fdocuments.net/reader034/viewer/2022042707/58e5765d1a28abbf5d8b4971/html5/thumbnails/7.jpg)
Meter Adapter
Gas Sensor
Electricity Sensor
Z-Wave
Phillips Hue
WiFi Router
Z-wave
Smart plugand smoke detectors
Central heating boiler
Boiler adaptor
Toon®
Toon connections - Data sources
![Page 8: Bigdata analytics and our IoT gateway](https://reader034.fdocuments.net/reader034/viewer/2022042707/58e5765d1a28abbf5d8b4971/html5/thumbnails/8.jpg)
The service center
Service Center
Toon Displays
Mobile devices
ProductApplication
s
Back OfficeApplication
s
Mobile Backend
Internet
Internet
VPNAccess
Data collector
![Page 9: Bigdata analytics and our IoT gateway](https://reader034.fdocuments.net/reader034/viewer/2022042707/58e5765d1a28abbf5d8b4971/html5/thumbnails/9.jpg)
Firstpartnership
![Page 10: Bigdata analytics and our IoT gateway](https://reader034.fdocuments.net/reader034/viewer/2022042707/58e5765d1a28abbf5d8b4971/html5/thumbnails/10.jpg)
Exponential growth
2013 30.0002014 90.0002015 190.0002016 300.0002017 (1.000.000)
2013 2014 2015 2016 2017
![Page 11: Bigdata analytics and our IoT gateway](https://reader034.fdocuments.net/reader034/viewer/2022042707/58e5765d1a28abbf5d8b4971/html5/thumbnails/11.jpg)
2 May 2023 12
Our journey towards bigdata 2014
• 30K customers • 40 types of
sensor data from 3K customers
• up to 1 minutes resolution
2012
• 4K customers • Aggregated
Energy data collected(DAY,WEEK,MONTH)
• Benchmarking functionality for customers
2013
• 90K customers • Data collection
from 30K customers
• Event and Measurement data
![Page 12: Bigdata analytics and our IoT gateway](https://reader034.fdocuments.net/reader034/viewer/2022042707/58e5765d1a28abbf5d8b4971/html5/thumbnails/12.jpg)
2 May 2023 13
Steps moving towards bigdata(cont’d)2017
• 300K customers • Near Real Time
aggregation with segmentation
• 400 signal/5 minutes
• Sharing Measurement data with tenants
2015
• 190K customers • Apache storm for
Near Real Time Energy consumption (3K customers)
• Data analysis using Cloudera stack
• Up to 10 seconds resolution
2016
• 330K customers• Collect data
from 230K customers anonymously
• Data as a service
• Analytics platform as a service
![Page 13: Bigdata analytics and our IoT gateway](https://reader034.fdocuments.net/reader034/viewer/2022042707/58e5765d1a28abbf5d8b4971/html5/thumbnails/13.jpg)
Toon®
Types of data
More than 300 types of sensor and user interaction data. If you want to know more please contact us : [email protected]
Energy CH-boiler (OT) Thermostat
Gas consumption
Electricity consumption
Water temperature
Service messages
Burner information
Room temperatureThermostat
programManual settings
Plug-dataElectricity
consumptionOn/off
Water pressure
Energy production
Energy feed-in
Boiler status
Touch eventsThermostat
Mobile
1 2 3 4 5
Thermostat setting
Hue data
![Page 14: Bigdata analytics and our IoT gateway](https://reader034.fdocuments.net/reader034/viewer/2022042707/58e5765d1a28abbf5d8b4971/html5/thumbnails/14.jpg)
2 May 2023 15
Anonymous Data Collection Service Center Customer
Always anonymous data collected
Customer may opt in for services and
send keys+ =
>Customer may opt out and
change keysData can’t be de-anonymized
again
![Page 15: Bigdata analytics and our IoT gateway](https://reader034.fdocuments.net/reader034/viewer/2022042707/58e5765d1a28abbf5d8b4971/html5/thumbnails/15.jpg)
2 May 2023 16
Sample Services with Toon’s Data?1. Benchmarking with daily energy meter data
![Page 16: Bigdata analytics and our IoT gateway](https://reader034.fdocuments.net/reader034/viewer/2022042707/58e5765d1a28abbf5d8b4971/html5/thumbnails/16.jpg)
2 May 2023 17
Sample Services with Toon’s Data? ( cont’d)2. Nearly Real Time Forecasting : Energy data Aggregation
![Page 17: Bigdata analytics and our IoT gateway](https://reader034.fdocuments.net/reader034/viewer/2022042707/58e5765d1a28abbf5d8b4971/html5/thumbnails/17.jpg)
2 May 2023 18
Sample Services with Toon’s Data? ( cont’d)3. Home Appliances Energy Consumption Detection: Energy Data disaggregation
![Page 18: Bigdata analytics and our IoT gateway](https://reader034.fdocuments.net/reader034/viewer/2022042707/58e5765d1a28abbf5d8b4971/html5/thumbnails/18.jpg)
2 May 2023 19
Sample Services with Toon’s Data ?(cont’d) 4. Boiler Maintenance Prediction
(KetelIQ)
![Page 19: Bigdata analytics and our IoT gateway](https://reader034.fdocuments.net/reader034/viewer/2022042707/58e5765d1a28abbf5d8b4971/html5/thumbnails/19.jpg)
2 May 2023 20
Sample Services with Toon’s Data? ( cont’d)5. Customer’s Behavioral studies
Click data Thermostat programs and co2 emission
![Page 20: Bigdata analytics and our IoT gateway](https://reader034.fdocuments.net/reader034/viewer/2022042707/58e5765d1a28abbf5d8b4971/html5/thumbnails/20.jpg)
21
What are we collecting? 24K measurements / day / customer 600KB (zipped) 230K customers
![Page 21: Bigdata analytics and our IoT gateway](https://reader034.fdocuments.net/reader034/viewer/2022042707/58e5765d1a28abbf5d8b4971/html5/thumbnails/21.jpg)
22
Arch. Principle : lambda Arch.
![Page 22: Bigdata analytics and our IoT gateway](https://reader034.fdocuments.net/reader034/viewer/2022042707/58e5765d1a28abbf5d8b4971/html5/thumbnails/22.jpg)
2 May 2023 23
Messaging
Data warehouse
Tools and Technologies
Analytic Clusters
Tools and Frameworks
Containers & micro services
Mon
itorin
g an
d al
ert
![Page 23: Bigdata analytics and our IoT gateway](https://reader034.fdocuments.net/reader034/viewer/2022042707/58e5765d1a28abbf5d8b4971/html5/thumbnails/23.jpg)
24
Data flow (partial view)
![Page 24: Bigdata analytics and our IoT gateway](https://reader034.fdocuments.net/reader034/viewer/2022042707/58e5765d1a28abbf5d8b4971/html5/thumbnails/24.jpg)
2 May 2023 27
Things to mention • ssl mutual authentication• raw-data stored partitioned in better way• Customers can control access to their own
data
![Page 25: Bigdata analytics and our IoT gateway](https://reader034.fdocuments.net/reader034/viewer/2022042707/58e5765d1a28abbf5d8b4971/html5/thumbnails/25.jpg)
2 May 2023 29
What are we doing with all that data?• Firehose – data as a service for B2B (and
researchers)• Analytic platform as a service• New data driven services for Toon users• Improve existing services• Marketing strategy (understand your customers)
![Page 26: Bigdata analytics and our IoT gateway](https://reader034.fdocuments.net/reader034/viewer/2022042707/58e5765d1a28abbf5d8b4971/html5/thumbnails/26.jpg)
2 May 2023 30
![Page 27: Bigdata analytics and our IoT gateway](https://reader034.fdocuments.net/reader034/viewer/2022042707/58e5765d1a28abbf5d8b4971/html5/thumbnails/27.jpg)
2 May 2023 31