Mesos and the Architecture of the New Datacenter

1

Mesos and the Architecture of the New Datacenter

@joerg_schad

Flink Cassandra Rails Spark memcached


A naive approach to handling varied app requirements: static partitioning.

This can cope with heterogeneity, but is very expensive.

KEEP IT STATIC

time


Maintaining sufficient headroom to handle peak workloads on all partitions leads to poor utilization overall.

KEEP IT STATIC

time


Multiple frameworks can use the same cluster resources, with their share adjusting dynamically.

SHARED RESOURCES

time

© Gerard Julien/AFP

Run everything in containers!

© 2015 Mesosphere, Inc. All Rights Reserved.

What about container management?

9


What about container management?

10


MESOS: ORIGINS


THE BIRTH OF MESOS

TWITTER TECH TALK

The grad students working on Mesos give a tech talk at Twitter.

March 2010

APACHE INCUBATION

Mesos enters the Apache Incubator.

Spring 2009

CS262B

Ben Hindman, Andy Konwinski and Matei Zaharia create “Nexus” as their

CS262B class project.

MESOS PUBLISHED

Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center is

published as a technical report.

September 2010

December 2010

DC/OS

April 2016


Sharing resources between batch processing frameworks

● Hadoop● MPI● Spark

What does an operating system provide?

● Resource management● Programming abstractions● Security● Monitoring, debugging, logging

TECHNOLOGY VISION


• A top-level Apache project• A cluster resource

negotiator• Scalable to 10,000s of

nodes• Fault-tolerant, battle-tested• An SDK for distributed apps• Native Docker support

14

Apache Mesos


ARCHITECTUREMESOS FUNDAMENTALS



● Agents advertise resources to Master● Master offers resources to Framework● Framework rejects/uses resources● Agents report task status to Master


YARN


Distributed Systems SDK„Developers will ask for

an API not for a VM to run their software“


● Focus on application logic, not on data-center structure

● Avoid networking-related code

● Reuse of built-in fault-tolerance and high availability

● Reuse distributed (infrastructure) frameworks (e.g., storage)

Distributed System SDKMesos


Scheduler Interface

● Utilise resources● ResourceOffer

● React on Tasks Events● TaskStatus Updates

Executor Interface (optional)

● Task life-cycle and monitoring● Provided executors

● Command Executor● Docker Executor

First Framework/ServiceMesos

27

WHERE DC/OS CAME FROM

PHYSICAL (x86) VIRTUAL HYPERSCALEMAINFRAME

SERVER VIRTUAL MACHINEPARTITION (LPAR)UNIT OF INTERACTION

● ERP, CRM, PRODUCTIVITY, MAIL & WEB SERVER

● LINUX, WINDOWS

● DATA / TRANSACTION PROCESSING

● UNIX, IBM OS/360

DEFINITIVE APPS AND OS

● ERP, CRM, PRODUCTIVITY, MAIL & WEB SERVER

● HYPERVISOR + GUEST OS

● BIG DATA, INTERNET OF THINGS, MOBILE APPS

● ???

???DATACENTER

NEW FORM FACTOR FOR DEVELOPING AND RUNNING APPS

● BIG DATA, INTERNET OF THINGS, MOBILE APPS

● DATACENTER OPERATING SYSTEM (DC/OS)

Datacenter Operating System (DC/OS)

Distributed Systems Kernel (Mesos)

DC/OS ENABLES MODERN DISTRIBUTED APPS

Big Data + Analytics EnginesMicroservices (in containers)

Streaming

Batch

Machine Learning

Analytics

Functions & Logic

Search

Time Series

SQL / NoSQL

Databases

Modern App Components

Distributed systems kernel to abstract resources

Ecosystem of frameworks & apps

Consistent architecture to run on top of kernel

User Interface (GUI & CLI)

Core system services (e.g., distributed init, cron, service discovery, package mgt & installer, storage)

Any Infrastructure (Physical, Virtual, Cloud)28

29

THEBASICS

DC/OS is … ● 100% open source (ASL2.0)

+ A big, diverse community● An umbrella for ~30 OSS projects

+ Roadmap and designs+ The build tool chain+ Docs and tutorials

● Not limited in any way● Familiar, with a few new features

31

THEUNIVERSE


Questions?

Thank you!

Mesos and the Architecture of the New Datacenter

Data & Analytics

Transcript of Mesos and the Architecture of the New Datacenter