with Kaa, Apache Cassandra, and Apache Zeppelin · PDF fileReal-time IoT data analytics and...

Click here to load reader

  • date post

    18-Aug-2018
  • Category

    Documents

  • view

    215
  • download

    0

Embed Size (px)

Transcript of with Kaa, Apache Cassandra, and Apache Zeppelin · PDF fileReal-time IoT data analytics and...

  • kaaproject.

    org 2016 CyberVision, Inc. All rights

    reserved.

    Real-time IoT data analytics and visualization with Kaa, Apache Cassandra,

    and Apache Zeppelin

  • Agenda

    Why Kaa? Why Cassandra? Why Zeppelin? Reference architecture overview Hands-on

    Sandbox setup Intel Edison application code walkthrough Cassandra appenders configuration Zeppelin configuration Live demo

    Q&A

    kaaproject.

    org 2016 CyberVision, Inc. All rights

    reserved.

  • About Kaa

    kaaproject.

    org 2016 CyberVision, Inc. All rights

    reserved.

  • Fully-featured IoT middleware platform 100% open-source (Apache license 2.0) Deploy anywhere: on-prem, in the cloud, in hybrid mode Horizontally, elastically scalable, and fault tolerant Guaranteed data delivery and reliable local storage Built-in transport security Efficient data serialization Rapid application development using C / C++ / Java / Objective C SDKs 10 Kb RAM footprint (with C SDK) Integration with popular device platforms

    Why Kaa?

    kaaproject.

    org 2016 CyberVision, Inc. All rights

    reserved.

  • Fault tolerant Performant Horizontally scalable Easy deployment Integration with popular analytics platforms

    Why Cassandra?

    kaaproject.

    org 2016 CyberVision, Inc. All rights

    reserved.

  • Multiple language backend Apache Spark integration Data visualization Dynamic forms Collaboration & Publish Open-source project

    Why Zeppelin?

    kaaproject.

    org 2016 CyberVision, Inc. All rights

    reserved.

  • Problem description

    Zone 1 Zone 2

    Zone 3 Zone 4

    Zone 5 Zone 6

    kaaproject.

    org 2016 CyberVision, Inc. All rights

    reserved.

  • Reference architecture

    Solar panels

    StructuredData

    Solar panels

    Raw data

    Intel Edison

    Kaa SDKClient application

    Intel Edison

    Kaa SDKClient application Kaa cluster/sandbox

    Kaa node

    Cassandra appender

    Cassandra

    Kaa node

    Cassandra appender

    Zeppelin instance/sandbox

    Zeppelin dashboard

    Zeppelin dashboard

    kaaproject.

    org 2016 CyberVision, Inc. All rights

    reserved.

  • Development environment setup

    Sample project repository: https://github.com/kaaproject/kaa-zeppelin-sample

    Kaa Sandbox: http://www.kaaproject.org/download-kaa

    Intel Edison:https://docs.kaaproject.org/display/KAA/Intel+Edison

    Zeppelin:http://zeppelin.incubator.apache.org

    kaaproject.

    org 2016 CyberVision, Inc. All rights

    reserved.

    https://github.com/kaaproject/kaa-zeppelin-samplehttps://github.com/kaaproject/kaa-zeppelin-samplehttps://github.com/kaaproject/kaa-zeppelin-samplehttps://github.com/kaaproject/kaa-zeppelin-samplehttps://github.com/kaaproject/kaa-zeppelin-samplehttp://www.kaaproject.org/download-kaa/http://www.kaaproject.org/download-kaa/https://docs.kaaproject.org/display/KAA/Intel+Edisonhttps://docs.kaaproject.org/display/KAA/Intel+Edisonhttp://zeppelin.incubator.apache.org/http://zeppelin.incubator.apache.org/

  • Data modeling

    Single zone per row Single panel per row with zone partitions

    Kaa data collection schema:{ "type":"record", "name":"PowerReport", "namespace":"org.kaaproject.kaa.sample", "fields":[ {"name":"timestamp", "type":"long"}, {"name":"zoneId", "type":{"type":"string", "avro.java.string":"String"}}, {"name":"panelId", "type":{"type":"string", "avro.java.string":"String"}}, {"name":"power", "type":"double"}]}

    kaaproject.

    org 2016 CyberVision, Inc. All rights

    reserved.

  • Single zone per row

    Partition Key: Zone ID Clustering Key: Timestamp, Panel ID

    Zone ID

    Timestamp, Panel ID

    Power

    Timestamp 2,Panel ID

    Power

    Timestamp N,Panel ID

    Power...

    Zeppelin query:%cassandra select zone_id, panel_id, power as "power, MW", timeFormat(timestamp) as time from logs.zone_per_row where zone_id={{zone_id='Sierra'}} and timestamp > logs.timeDelta(toUnixTimestamp(now()), 60000) ORDER BY timestamp ASC;

    kaaproject.

    org 2016 CyberVision, Inc. All rights

    reserved.

  • Single panel per row with zone partitions

    Partition Key: Zone ID, Panel ID Clustering Key: Timestamp

    Zone ID,Panel ID

    Timestamp N

    Power

    Timestamp N-1

    Power

    Timestamp 1

    Power...

    kaaproject.

    org 2016 CyberVision, Inc. All rights

    reserved.

    Zeppelin query:%cassandra select zone_id, panel_id, power as "power, MW", timeFormat(timestamp) as time from logs.panel_per_row where zone_id={{zone_id='Sierra'}} and panel_id={{panel_id='Panel_0'}} and timestamp > logs.tmBefore(toUnixTimestamp(now()), 60000) ORDER BY timestamp ASC;

  • kaaproject.orgcybervisiontech.com

    THANK YOU FOR YOUR ATTENTIONQUESTIONS?

  • Fault-tolerance and horizontal scalability

    kaaproject.org

    Operations servers

    2016 CyberVision, Inc. All rights reserved.

    Bootstrap servers

    Zookeeper quorum

    Endpoints (Kaa SDK embedded)

    Control servers

    standby

    active