#OSSPARIS17 - PrestoCloud Project: Big Data Orchestration for Batch+Stream in Clouds&Fogs with...

15
Paris, Sophia Antipolis, London, San Jose USA PrestoCloud: Big Data Orchestration for Batch+Stream in Clouds & Fogs with ActiveEon Denis Caromel, Vincent Kherbache, Sophie Song, Pascal Urso

Transcript of #OSSPARIS17 - PrestoCloud Project: Big Data Orchestration for Batch+Stream in Clouds&Fogs with...

Paris, Sophia Antipolis, London, San Jose USA

PrestoCloud: Big Data Orchestration for

Batch+Stream in Clouds & Fogs with ActiveEon

Denis Caromel, Vincent Kherbache, Sophie Song, Pascal Urso

Global Locations

Partnerships

Key information

Management

Denis

Caromel, CEO

François

Tournesac, CSO

Brian Amedro,

CTO

Founded in 2007 by Denis Caromel in Sophia-Antipolis, Spin-off of INRIA

Addressing $80 Billion Hybrid Cloud Market with 27% CAGR

Disruptive Patented Technology w/ Exceptional Business Outcomes

60% of the revenue from international

Sophia-Antipolis (France)

Paris (France)

London (United Kingdom)

San-Jose (United States)

Montreal (Canada)

Fribourg (Switzerland)

Dakar (Senegal)

ProActive Solution

Job Scheduling, Workload Automation

Orchestration & Meta-Scheduling

On-premises and on all clouds

Open Source

Core Product

Processing & Automation Workflows

Scheduling & Orchestration

Resource

Management

& Monitoring

Workflow

Studio

Job

Console

Resource

Manager

Big Compute for Everyone

</>

Remote

Visualization

Integrated Web

PortalsAPI

Scheduler & Meta-Scheduler

Resource Manager

Clusters

LSF

Application-specific

Interfaces

ProActive Workflows

Fault Tolerance

Cloud bursting

Resource agnostic

Micro-service

PrEstoCloud

Proactive Cloud Resources Management At The Edge For Efficient Real-time Big Data Processing

PrEstoCloud Factsheet

Partners

• 36 months • Starts on January 2017

• Ends on December 2019

• ActiveEon: • Core tech Provider

• 60 PM

• 501K€

• Provides Cloud Automation, Workflows and Scheduling for Real-Time Big Data Processing

Role of Partners

PrEstoCloudPrEstoCloud Platform

Cloud-Edge Communication Layer

Cloud Application

Communication & Message Broker

Devices Layer

ApplicationDeveloper

Control Layer

Code-level Annotation

s

Application Requirements

Meta-Management Layer

Semantic Model

DevOps placement requirements

scalability requirements

Events & Messages

Mesh Network-aware application

Spatio- Temporal Processing

Library

Resources Adaptation

Recommender

Application Fragmentation &

Deployment Recommender

Situation Detection

Mechanism

Workload predictor

Mobile Context Analyzer

predictions

Mesh Network-aware application

Spatio- Temporal Processing

Library

Security Enforcement Mechanism

Cloud Infrastructure Layer

Events & Messages

Application Placement & Scheduling

Controller

Autonomic ResourceManager

Autonomic Data Intensive Application

Manager

DevOps

Cloud Deployment and Management

Edge Deployment and Management

Monitoring Probes

On/Offloading Agents

Monitoring Probes

On/Offloading Agents

Monitoring Probes

Inter-site Network Virtualisation

Mobile On/Offload Processing

Cloud and Edge

Resources Repository

Events & Messages

Real Time Data Streams

Edge application

Monitoring Probes

On/Offloading Agents

Events & Messages

Mesh Network-aware application

Spatio- Temporal Processing

Library

Monitoring Probes

On/Offloading Agents

Real Time Data Streams

Monitoring Probes Installer

8

Resources Management

Extended TOSCA specification

• On public & private clouds: Virtual machines, Containers

• On edge devices: Containers, Mobile app

Inter-site, multi-layer network virtualization

• Route-based VPN

• Security enforcement mechanism (RBAC, NFV)

Real-time big data processing

Resource provisioning

• Scheduling and orchestration of (de)commissioning workflows

• Real-time events for the monitoring of Cloud/Edge resources and applications

Message broker (AMQP, MQTT, etc.)

• Decision-making in the resources placement

Recommendation system (Constraints solver)

Data-intensive applications management

• Predictive Workload management

• Scheduling and orchestration of deployment workflows

(Re)configuration

Kafka Storm Visdom Streaming

Real-Time Data Streaming using ActiveEonWorkflows:

• Apache Kafka (Data streaming)

• Apache Storm (Real-time stream processing)

• Visdom (Stream Visualization from Facebook)

https://www.youtube.com/watch?v=i6ZOyoXSq-A&t=5s

Testing the NewAzure Data Centerswith Microsoft & ActiveEon

Azure PoC in the Box

Azure Node Source InfrastructureAzure Node Source Infrastructure

Scale automaticallyLeverage Azure services

Existing Resources

Local & Network Resources Private Cloud, HPC & Others

Resource

Manager

Workflow

Scheduler

</>

Azure Node Source Infrastructure

LSF

Using Azure Scale Sets

Deploy over

150 k nodes

150 K Nodes Benchmarks on Azure

Paris, Sophia Antipolis, London, San Jose USA @activeeon

[email protected]

+33 988 777 660

Automate Accelerate & Scale

10K Nodes, 20K Tasks, 1M Jobs