Building a Data-Friendly Platform for a Data-Driven Future

© 2016 Mesosphere, Inc. All Rights Reserved.

Building a Data-Friendly Platform for a Data-Driven Future

Benjamin Hindman - @benh

© 2015 Mesosphere, Inc. All Rights Reserved. 2

$ whoamiINTRO

BENJAMIN HINDMAN

Co-founder and Chief Architect of Mesosphere, Inc.

Formerly Twitter, UC Berkeley

Co-creator of Apache Mesos

@benh


REINFORCING TRENDS

microservices containerization

container/cluster/resource management


REINFORCING TRENDS




REINFORCING TRENDS



big data & analytics


TWITTERMONORAIL


TWITTERSERVICES


PAINS FROM THE MONOLITHIC ARCHITECTURE


MONOLITHIC TO MICROSERVICES


1.Do one thing and do it well (UNIX).

2.Compose!

3.Test and debug in isolation.

4.Captures organizational structure (many teams working in parallel).

TREND TOWARDS MICROSERVICES


MICROSERVICES

Traditional Application Architecture Today’s Microservices Application Architecture

REST

API

s

Hard to scale,wasting resources

Many functions in a single process

Cross-functional teams organized around capabilities

Scalable, efficient and fully dynamic

Siloed teams

Each element of functionality defined as “microservices”


CONTAINERIZATION

then now


CONTAINERIZATION

then nowmore moving parts less moving parts


ANATOMY OF MODERN APPLICATIONS

Data StorageBig Data Processing

X

X X...

..

.Message Queue

..

.... X

Anything else

Functions & Logic Big Data & Analytics

Microservicesin containers

Distributed Systems


Microservices, containers, and big data lead to more difficult operations w/out proper infrastructure.

Challenges:

1)Deployment and scheduling for fault-tolerance.

2)Scheduling for elasticity and efficiency.

3)Service discovery (“naming”) and networking.

4)Operations: maintenance and upgrades.

5)…

NEED FOR NEW INFRASTRUCTURE


CHALLENGES

FAILURES


MAINTENANCECHALLENGES


BREAK OUT OF TRADITIONAL INFRASTRUCTURE SILOS

• Many silos.

• Management nightmare.

• Lengthy cycles to deploy code.• Low utilization.

Apache Mesos and the DC/OS

TRADITIONAL APPROACH

PaaS 1

ContainerApp

1Big Data

Analytics 1Big Data

Analytics 2

PaaS 2

ContainerApp

2Stateful Service

1

Stateful Service

2

UNIFIED APPROACH

ContainerApps(All) Big Data Analytics

(All)

PaaS (All)

Stateful Service(All)

• High performance and resource isolation.

• Easy scalability and multi-tenancy.

• Fault tolerant and highly available.• Highly efficient with highest utilization.

Deploy on-prem or in cloud


RUN EVERYTHING ON THE SAME SHARED INFRASTRUCTUREUTILIZATION

siloed, over-provisioned servers,low utilization

Industry Average12-15% utilization


RUN EVERYTHING ON THE SAME SHARED INFRASTRUCTUREUTILIZATION

Siloed, over-provisioned servers,low utilization

Automated schedulers, workload multiplexing, less machines or more applications

Industry Average12-15% utilization

30-40% utilization, up to 96% for some users


EVOLUTION: FROM STATIC TO DYNAMIC INFRASTRUCTURE

MAINFRAMEData / Transaction

Processing

PHYSICAL (x86)Client-Server Apps

(ERP, CRM, Productivity)

VIRTUALWeb-Based Applications(Enterprise & Consumer)

Existing Computing Infrastructure Is Inefficient And Not Suitable for Modern Workloads

Siloed, Static, Monolithic & Manual

t

Computing Infrastructure Evolution

Efficient, Dynamic, Agile & Automated

UNIFIEDStateless Microservices

Stateful Distributed Systems Analytics


THE DATACENTER COMPUTER


1. TREAT MACHINES AS CATTLE NOT PETS

Keep the base operating system small and simple, run “containerized” applications.

2. AUTOMATE WITH SOFTWARE NOT HUMANS

Let software schedule software, i.e., handle failures, improve utilization, and manage maintenance.

DATACENTER COMPUTER PRINCIPLES

desktop computer

server datacenter

OS

OS

OS

Datacenter Computer Needs an Operating System

Distributed Systems Kernel (Mesos)

DATACENTER OPERATING SYSTEM (DC/OS)

Distributed systems kernel to abstract resources

26




User Interface (GUI & CLI)

Core system services (e.g., distributed init, cron, service discovery, package mgt & installer, storage)

27

Datacenter Operating System (DC/OS)




User Interface (GUI & CLI)

Core system services (e.g., distributed init, cron, service discovery, package mgt & installer, storage)

28

Datacenter Operating System (DC/OS)

On Premise AWS Azure


Apache Mesos is a general purpose cluster manager (i.e., not just focused on batch computation).

WHAT IS APACHE MESOS?MESOS


CLUSTER MANAGEMENTSOLUTIONS | ACADEMIA VS INDUSTRY

IndustryAcademia


DIFFERENT SOFTWARESOLUTIONS | ACADEMIA VS INDUSTRY

IndustryAcademia

● MPI (Message Passing Interface) ● Apache (mod_perl, mod_php)● Web Service (Java, Ruby, …)


DIFFERENT SCALE (AT FIRST)SOLUTIONS | ACADEMIA VS INDUSTRY

IndustryAcademia

● 100’s of machines ● 10’s of machines



IndustryAcademia

● PBS (Portable Batch System)● TORQUE● SGE (Sun Grid Engine)

● SSH● Puppet / Chef● Capistrano / Ansible

Cluster Managers


DIFFERENT SCALE (CONVERGING)SOLUTIONS | ACADEMIA VS INDUSTRY

IndustryAcademia

● 100’s of machines ● 10’s of machines

1,000s of machines



IndustryAcademia

● PBS (Portable Batch System)● TORQUE● SGE (Sun Grid Engine)

● SSH● Puppet / Chef● Capistrano / Ansible

Batch Computation!

Mesos is a cluster manager with a master/agent architecture

masters

agents


master

agents

coordinator coordinatorcoordinator

38

MESOS

2-LEVEL SCHEDULING

Mesos challengedthe status quoof cluster managers

cluster manager status quo

cluster manager

application

specificatio

n

the specification includes as much

information as possible to assist

the cluster manager in scheduling

and execution


cluster manager

application wait for task

to be

executed


cluster manager

application

resul

t

problems with specifications

� hard to specify certain desires or constraints

� hard to update specifications dynamically as tasks execute

and finish/fail

the bigger picturecluster manager has inadequate knowledge of distributed

system’s execution needs/semantics to make optimal decisions

distributed system’s execution needs/semantics can’t easily or efficiently be expressed to cluster manager

MapReduce specification?


fine-grainedcourse-grained


best resource utilization, but

hard if impossible to specify




worst resource utilization,

but easy to express (how

most cluster managers run

something like Hadoop)

distributed systems register with the Mesos master(s) in order to run computations

masters

agents

frameworks

Mesos model

masters

coordinator

request

3 CPUs

2 GB RAM

a request is purposely simplified

subset of a specification including

just the required resources

at that point in time

question: what should you do if you can’t satisfy a request?


� wait until you can …


� wait until you can …

� offer best you can immediately

Mesos model

masters

coordinator

offer

hostname

4 CPUs

4 GB RAM

offer

hostname

4 CPUs

4 GB RAM

offer

hostname

4 CPUs

4 GB RAM

offer

hostname

4 CPUs

4 GB RAM

Mesos model

masters

coordinator

offer

hostname

4 CPUs

4 GB RAM

offer

hostname

4 CPUs

4 GB RAM

offer

hostname

4 CPUs

4 GB RAM

offer

hostname

4 CPUs

4 GB RAM

Mesos model

masters

coordinator

offer

hostname

4 CPUs

4 GB RAM

distributed system uses the

offers to perform it’s own

scheduling

Mesos model

masters

coordinator



scheduling

task

3 CPUs

2 GB RAM

Mesos model

masters

coordinator



scheduling

multi-level scheduling

task

3 CPUs

2 GB RAM

1st level: master makes allocations via offers

2nd level: distributed systems schedule tasks using offers


MESOS

LEVEL OF INDIRECTION

Mesos (agents)

coordinator

Mesos (master)

coordinator

responsible for allocation (and reallocation) of resources


DATACENTER KERNELMESOS

provides common functionality every new distributed system re-implements:

• resource allocation• resource deallocation• resource reservations• resource isolation• resource monitoring• failure detection• package distribution• task starting, killing, cleanup• volume management• …

today tomorrow

YOU

don’treinventthewheel!

distributed systemsare hard to build

distributed systemsare hard to operate: deploy, maintain, update

operating distributed systems

operating distributed systems

� download

� deploy (read book; HA? arrangement?)

� monitor (read book; logs, metrics, alerting)

� maintain� debug (read book, ask internet, IRC)

� “fix” problem (update runbooks, write scripts,

build ancillary system!)

� upgrade (and/or redeploy)

perspectivedistributed systems should be able to operate themselves:

deploy, monitor, update, upgrade, etc

need an interface that enables distributed systems to

communicate with the underlying infrastructure, and vice versa

perspectivedistributed systems should be able to operatethemselves: deploy, monitor, update, upgrade,

etc

need an interface that enables distributed

systems to communicate with the underlying

infrastructure, and vice versa

Apache Mesos

why: the bigger picturehumans have inadequate knowledge of distributed system

needs/semantics to make optimal decisions (even after reading

the book)

distributed system execution needs/semantics can’t easily or efficiently be expressed to humans and/or underlying

infrastructure

Linuxapplications “operate” themselves on Linux, when they need

more CPU, they ask the Linux kernel for more CPUs (processes,

threads); when they need certain operations to be performed

(e.g., write/read to file), they ask the Linux kernel; …

Mesosdistributed systems “operate” themselves on Mesos, when they

need more CPU, they ask Mesos for more CPUs (tasks,

containers); when they need certain operations to be performed

(e.g., create a persistent volume), they ask Mesos; …


AUTOMATED OPERATIONS OF DISTRIBUTED SYSTEMS

Software will manage itself, using Mesos and the DC/OS API

● Most distributed systems are difficult to manage but they don’t need to be.

Kafka Spark

Cassandra

Data processing engine

Messaging backbone

Distributed database

HDFSDistributed file

system

73

THEUNIVERSE


TWEETER ARCHITECTURE ON DC/OS

75

CASE STUDY

Forging Ahead with Mesos, Containers and DC/OS

Having now run our event streaming and big data ingestion pipeline services in production on DC/OS, across 3 regions, over the last year, we've achieved the following results:

● A 66% reduction in AWS Instances

● Cost Improvements up to 57%

● An impressive 40 sec time to deploy a new build with zero downtime

● A 3 min time to stand up a new region

● 100% Uptime

● Total Resources needed: 1 DevOps Engineer


Check outdcos.io

THANK YOU

Building a Data-Friendly Platform for a Data-Driven Future

Documents

Transcript of Building a Data-Friendly Platform for a Data-Driven Future