Apache Ignite and Apache Spark - GridGain Systems · Ignite and Spark Integration Spark Application...

17
© 2017 GridGain Systems, Inc. Where Fast Data Meets the IoT Apache Ignite and Apache Spark Denis Magda Ignite PMC Chair GridGain PM

Transcript of Apache Ignite and Apache Spark - GridGain Systems · Ignite and Spark Integration Spark Application...

Page 1: Apache Ignite and Apache Spark - GridGain Systems · Ignite and Spark Integration Spark Application Spark Worker Spark Job Spark Job Yarn Mesos Docker HDFS Spark Worker Spark Job

© 2017 GridGain Systems, Inc.

Where Fast Data Meets the IoT Apache Ignite and Apache Spark

Denis Magda Ignite PMC Chair GridGain PM

Page 2: Apache Ignite and Apache Spark - GridGain Systems · Ignite and Spark Integration Spark Application Spark Worker Spark Job Spark Job Yarn Mesos Docker HDFS Spark Worker Spark Job

© 2017 GridGain Systems, Inc.

• IoT Demands to Software • IoT Software Stack • Device OS/RTOS • Data Collection and Enrichment • HTAP Platform • Application APIs

• Demo

Agenda

Page 3: Apache Ignite and Apache Spark - GridGain Systems · Ignite and Spark Integration Spark Application Spark Worker Spark Job Spark Job Yarn Mesos Docker HDFS Spark Worker Spark Job

© 2017 GridGain Systems, Inc.

IoT Demands to Software

Real-time Processing

SQL, Geo-Spatial

Analytics (BI, ML)

High-Availability

Simple Scalability

Page 4: Apache Ignite and Apache Spark - GridGain Systems · Ignite and Spark Integration Spark Application Spark Worker Spark Job Spark Job Yarn Mesos Docker HDFS Spark Worker Spark Job

© 2017 GridGain Systems, Inc.

IoT Software Stack

Device OS/Real-Time OS

Data Collection and Enrichment

HTAP Platform

Application APIs

Page 5: Apache Ignite and Apache Spark - GridGain Systems · Ignite and Spark Integration Spark Application Spark Worker Spark Job Spark Job Yarn Mesos Docker HDFS Spark Worker Spark Job

© 2017 GridGain Systems, Inc.

Apache IoT Software Stack

Device OS/Real-Time OS

Data Collection and Enrichment

HTAP Platform

Application APIs

Page 6: Apache Ignite and Apache Spark - GridGain Systems · Ignite and Spark Integration Spark Application Spark Worker Spark Job Spark Job Yarn Mesos Docker HDFS Spark Worker Spark Job

© 2017 GridGain Systems, Inc.

Apache MyNewt

Open Source RTOS Cortex M, MIPS Bluetooth, Wifi,

TCP/IP

Secure Bootloader

Remote Firmware Upgrade

Page 7: Apache Ignite and Apache Spark - GridGain Systems · Ignite and Spark Integration Spark Application Spark Worker Spark Job Spark Job Yarn Mesos Docker HDFS Spark Worker Spark Job

© 2017 GridGain Systems, Inc.

Data Collection and Enrichment

DURABLE MEMORY

DURABLE MEMORY

Ignite Cluster

Page 8: Apache Ignite and Apache Spark - GridGain Systems · Ignite and Spark Integration Spark Application Spark Worker Spark Job Spark Job Yarn Mesos Docker HDFS Spark Worker Spark Job

© 2017 GridGain Systems, Inc.

Apache Ignite In-Memory Computing Platform

Memory-Centric Storage

Ignite Native Persistence (Flash, SSD, Intel 3D XPoint)

Third-Party Persistence (RDBMS, HDFS, NoSQL)

SQL Transactions Compute Services MLStreaming

Applications

Key/Value

IoTFinancial Services

Pharma & Healthcare

E-CommerceTravel & Logistics

Telco

Page 9: Apache Ignite and Apache Spark - GridGain Systems · Ignite and Spark Integration Spark Application Spark Worker Spark Job Spark Job Yarn Mesos Docker HDFS Spark Worker Spark Job

© 2017 GridGain Systems, Inc.

Data Grid

JCache Transactions Compute SQL

RDBMS

NoSQLHDFS

Server Node

Distributed Key-Value Store

Dynamic Scaling

Distributed partitioned hash map

ACID TransactionJCache & SQL

Server Node Server Node

3rd party storage caching

DURABLE MEMORY DURABLE MEMORY DURABLE MEMORY

Page 10: Apache Ignite and Apache Spark - GridGain Systems · Ignite and Spark Integration Spark Application Spark Worker Spark Job Spark Job Yarn Mesos Docker HDFS Spark Worker Spark Job

© 2017 GridGain Systems, Inc.

Distributed SQL

JDBC ODBC SQL API

Java .NET C++ BI

SELECT, UPDATE, INSERT, MERGE, DELETE, CREATE and ALTER

DDL, DML Support

Cross-platform Compatibility

Indexes in RAM or Disk

Dynamic Scaling

Server Node Server NodeServer Node

Apache Ignite Cluster

DURABLE MEMORY DURABLE MEMORY DURABLE MEMORY

Tools

Page 11: Apache Ignite and Apache Spark - GridGain Systems · Ignite and Spark Integration Spark Application Spark Worker Spark Job Spark Job Yarn Mesos Docker HDFS Spark Worker Spark Job

© 2017 GridGain Systems, Inc.

Compute Grid

DURABLE MEMORY

DURABLE MEMORY

Ignite Cluster

C1

R1

C2

R2

C = C1 + C2

R = R1 + R2

C = Compute

R = Resultin T/2 time

Automatic Failover

Load Balancing

Zero Deployment

Page 12: Apache Ignite and Apache Spark - GridGain Systems · Ignite and Spark Integration Spark Application Spark Worker Spark Job Spark Job Yarn Mesos Docker HDFS Spark Worker Spark Job

© 2017 GridGain Systems, Inc.

Ignite and Spark Integration

Spark Application

Spark Worker

Spark Job

Spark Job

Yarn Mesos Docker HDFS

Spark Worker

Spark Job

Spark Job

Spark Worker

Spark Job

Spark Job

In-Memory Shared RDD or DataFrame

Share RDD across jobs on

the hostIn-Memory Indexes

SQL on top of RDDs

Share RDD Globally

Ignite Node Ignite Node Ignite Node

Page 13: Apache Ignite and Apache Spark - GridGain Systems · Ignite and Spark Integration Spark Application Spark Worker Spark Job Spark Job Yarn Mesos Docker HDFS Spark Worker Spark Job

© 2017 GridGain Systems, Inc.

Machine Learning Grid

K-Means Regressions Decision Trees

R C++ Python Java

Server Node Server NodeServer Node

Distributed Core Algebra

DURABLE MEMORY DURABLE MEMORY DURABLE MEMORY

Scala REST

Random ForestDistributed Algorithms

Dense and Sparse Algebra

Collocated Processing

Multi-Language Support

Page 14: Apache Ignite and Apache Spark - GridGain Systems · Ignite and Spark Integration Spark Application Spark Worker Spark Job Spark Job Yarn Mesos Docker HDFS Spark Worker Spark Job

© 2017 GridGain Systems, Inc.

Apache Ignite Users

FinTech

Financial Services Software Logistics & Travel

E-commerce

Telco

IoT

Pharma & HealthcareAdtech

Page 15: Apache Ignite and Apache Spark - GridGain Systems · Ignite and Spark Integration Spark Application Spark Worker Spark Job Spark Job Yarn Mesos Docker HDFS Spark Worker Spark Job

© 2017 GridGain Systems, Inc.

The company develops IoT solutions that transmit energy consumption data between meters, consumers and utilities in real time.

Problem • Could not meet latency and throughput SLAs • Missing scalability and elasticity

GridGain Solution • 50 millions meters stream the data back in real-time • Collocated in-memory processing • Advanced security and multi-tenancy

SQL

Smart Meters

GridGain Ignite Cluster

DB

IN-MEMORY IN-MEMORY IN-MEMORY IN-MEMORY

GridGain Advanced Security

Large IoT Provider - Smart Metering and Utilities

Compute Transactions

Company’s Platform

Page 16: Apache Ignite and Apache Spark - GridGain Systems · Ignite and Spark Integration Spark Application Spark Worker Spark Job Spark Job Yarn Mesos Docker HDFS Spark Worker Spark Job

© 2017 GridGain Systems, Inc.

Demo

Page 17: Apache Ignite and Apache Spark - GridGain Systems · Ignite and Spark Integration Spark Application Spark Worker Spark Job Spark Job Yarn Mesos Docker HDFS Spark Worker Spark Job

© 2017 GridGain Systems, Inc.

Thank you for joining us. Follow the conversation.http://ignite.apache.org

Any Questions?

#apacheignite #denismagda