Apache Flink Meets Apache Mesos And DC/OS @ Mesos Meetup Berlin

31
Till Rohrmann [email protected] @stsffap Apache Flink® Meets Apache Mesos® and DC/OS

Transcript of Apache Flink Meets Apache Mesos And DC/OS @ Mesos Meetup Berlin

Page 1: Apache Flink Meets Apache Mesos And DC/OS @ Mesos Meetup Berlin

Till Rohrmann [email protected] @stsffap

Apache Flink® Meets Apache Mesos® and DC/OS

Page 2: Apache Flink Meets Apache Mesos And DC/OS @ Mesos Meetup Berlin
Page 3: Apache Flink Meets Apache Mesos And DC/OS @ Mesos Meetup Berlin
Page 4: Apache Flink Meets Apache Mesos And DC/OS @ Mesos Meetup Berlin

MapReduce is crunching Data

Page 5: Apache Flink Meets Apache Mesos And DC/OS @ Mesos Meetup Berlin

We need to turn faster!

Page 6: Apache Flink Meets Apache Mesos And DC/OS @ Mesos Meetup Berlin

FMACK Stack

EVENTS

Ubiquitous data streams from

connected devices

INGEST

Apache Kafka

STORE

Apache Flink

ANALYZE

Apache Cassandra

ACT

Akka

Ingest millions of events per second

Distributed & highly scalable database

Real-time and batch process data

Visualize data & build data driven apps

Mesos/ DC/OS

Sensors

Devices

Clients

Page 7: Apache Flink Meets Apache Mesos And DC/OS @ Mesos Meetup Berlin

Datacenter

Page 8: Apache Flink Meets Apache Mesos And DC/OS @ Mesos Meetup Berlin

Naive Approach

Typical Datacentersiloed, over-provisioned servers,

low utilization

Industry Average12-15% utilization

mySQL

microservice

Cassandra

Flink

Kafka

Page 9: Apache Flink Meets Apache Mesos And DC/OS @ Mesos Meetup Berlin

© 2017 Mesosphere, Inc. All Rights Reserved. 9

Page 10: Apache Flink Meets Apache Mesos And DC/OS @ Mesos Meetup Berlin

Apache Mesos

Typical Datacentersiloed, over-provisioned servers,

low utilization

Industry Average12-15% utilization

mySQL

microservice

Cassandra

Flink

Kafka

Mesosautomated schedulers, workload multiplexing

onto the same machines

Page 11: Apache Flink Meets Apache Mesos And DC/OS @ Mesos Meetup Berlin
Page 12: Apache Flink Meets Apache Mesos And DC/OS @ Mesos Meetup Berlin

Original creators of Apache Flink®

Providers of the dA Platform, a supported

Flink distribution

Page 13: Apache Flink Meets Apache Mesos And DC/OS @ Mesos Meetup Berlin

Apache Flink in a Nutshell

Event-driven applications (event sourcing, CQRS)

Stateful, event-driven,event-time-aware processing

Batch Processing (data sets)

Stream Processing / Analytics (data streams, windows, …)

Page 14: Apache Flink Meets Apache Mesos And DC/OS @ Mesos Meetup Berlin

Programming Model

Computation

Computation

Computation

Computation

Source Source

SinkSink

Transformation

state

state

state

state

Page 15: Apache Flink Meets Apache Mesos And DC/OS @ Mesos Meetup Berlin

What is Flink Good For?

Page 16: Apache Flink Meets Apache Mesos And DC/OS @ Mesos Meetup Berlin

Detecting fraud in real time

As fraudsters get better, need to update models without downtime

Live 24/7 service

Credit card transactions

Notifications and alerts

Evolving fraud models built by data scientists

@

Page 17: Apache Flink Meets Apache Mesos And DC/OS @ Mesos Meetup Berlin

▪ Athena X (https://eng.uber.com/athenax/) ▪ Streaming analytics platform ▪ SQL as abstraction layer

Streams from Hadoop, Kafka, etc

SQL, thresholds, actions

Analytics Alerts

Derived streams

@

Page 18: Apache Flink Meets Apache Mesos And DC/OS @ Mesos Meetup Berlin

▪ Blink based on Flink ▪ A core system in Alibaba Search

• Machine learning, search, recommendations • A/B testing of search algorithms • Online feature updates to boost conversion rate

@

Page 19: Apache Flink Meets Apache Mesos And DC/OS @ Mesos Meetup Berlin

@

Complete social network Implemented using event sourcing andCQRS (Command Query Responsibility Segregation) https://data-artisans.com/blog/drivetribe-cqrs-apache-flink

Page 20: Apache Flink Meets Apache Mesos And DC/OS @ Mesos Meetup Berlin

Apache Flink & Apache Mesos

Page 21: Apache Flink Meets Apache Mesos And DC/OS @ Mesos Meetup Berlin

Why Apache Mesos?▪ Mesos offers full functionality to implement fault

tolerant and elastic distributed applications

▪ 30% of survey respondents were running Flink on Mesos (prior to proper Mesos support, September 2016)

Page 22: Apache Flink Meets Apache Mesos And DC/OS @ Mesos Meetup Berlin

Flink’s Mesos Integration

Apache Flink FrameworkMesos Master

Mesos App Master

Flink MesosResourceManager

JobManager

Mesos Task

TaskManager

Mesos Task

TaskManager

Allocate Resources

Launch Mesos tasks

Register

Execute Job

Page 23: Apache Flink Meets Apache Mesos And DC/OS @ Mesos Meetup Berlin

Resource Manager Components

▪ Monitors connection to Mesos

Connection Monitor Launch Coordinator

▪ Resource offer processing and task scheduling

▪ Gathers offers and matches them to tasks using Fenzo

Task MonitorReconciliation Coordinator

▪ Monitors Mesos tasks ▪ Triggers reconciliation ▪ Makes sure tasks are properly killed

▪ Reconciles tasks view between ResourceManager and Mesos Master

Page 24: Apache Flink Meets Apache Mesos And DC/OS @ Mesos Meetup Berlin

Component Interplay

ResourceManager

Connection Monitor

Launch Coordinator

Task MonitorReconciliation Coordinator

Mesos MasterResource offers

Launch tasks

Monitor tasks

Status messages

Trigger reconciliation

Status messages

Mesos Task

Reconcile tasks

Start TaskManagers

Recover tasks

Kill task

Page 25: Apache Flink Meets Apache Mesos And DC/OS @ Mesos Meetup Berlin
Page 26: Apache Flink Meets Apache Mesos And DC/OS @ Mesos Meetup Berlin

DC/OS

Datacenter Operating System (DC/OS)

Distributed Systems Kernel (Mesos)

Big Data + Analytics EnginesMicroservices (in containers)

Streaming

Batch

Machine Learning

Analytics

Functions & Logic

Search

Time Series

SQL / NoSQL

Databases

Modern App Components

Any Infrastructure (Physical, Virtual, Cloud)

Page 27: Apache Flink Meets Apache Mesos And DC/OS @ Mesos Meetup Berlin

Demo Time

Generator

▪ Financial data generated by generator ▪ Written to Kafka topics ▪ Kafka topics consumed by Flink ▪ Flink pipeline operates on Kafka data ▪ Results written back into Kafka

Page 28: Apache Flink Meets Apache Mesos And DC/OS @ Mesos Meetup Berlin

Conclusion

Page 29: Apache Flink Meets Apache Mesos And DC/OS @ Mesos Meetup Berlin

TL;DL

▪ Apache Flink modern stream processor for real-time processing and event-driven applications

▪ Apache Flink runs on Mesos using Fenzo

▪ DC/OS offers easy to use Flink package

Page 30: Apache Flink Meets Apache Mesos And DC/OS @ Mesos Meetup Berlin

30

Thank you!@stsffap @ApacheFlink @dataArtisans

Page 31: Apache Flink Meets Apache Mesos And DC/OS @ Mesos Meetup Berlin

We are hiring!

data-artisans.com/careers