Mesos

Benjamin Hindman @benhDatacenter Management with Apache Mesosmesos.apache.org@ApacheMesos

1

Ive got tons of data ...

more everyday!

That must be why they call it a datacenter.

Id love to answer some questions with the help of my data!

I think Ill try Hadoop.your datacenter

+ Hadoop

happy?

Not exactly

Hadoop is a big hammer, but not everything is a nail!

Ive got some iterative algorithms, I want to try Spark!datacenter management

datacenter management

datacenter management

static partitioning

Oh noes! Spark wants to read and write data to HDFS! Hadoop

(map/reduce)(distributed file system)HDFS

HDFS

Could we just give Spark its own HDFS cluster too?

HDFS

HDFS

HDFS

HDFS

tee incoming data(2 copies)HDFS

tee incoming data(2 copies)periodic copy/sync

That sounds annoying lets not do that. Can we do any better though?HDFS

HDFS

HDFS

happy now?No! Weve decided to start doing real time computation with Storm

datacenter management

datacenter management

happy now!?

Not really during the day Id rather give more machines to Spark but at night Id rather give more machines to Hadoop!datacenter management

datacenter management

datacenter management

datacenter management

And failures require more datacenter management! datacenter management

datacenter management

datacenter management

I dont want to deal with this!the datacenter rather than think about the datacenter like this

is a computerthink about it like this

datacenter computer

applicationsresourcesfilesystemmesos

applicationsresourcesfilesystemkernel

Okay, so how does it work?Step 1: HDFS

Step 2: Mesosrun a master (or multiple for high availability)

Step 2: Mesosrun slaves on the rest of the machines

Step 3: Frameworks

Step 3: Frameworks

Step 3: Frameworks

Step 3: Frameworks

Step 3: Frameworks

Step 3: Frameworks

Step 3: Frameworks

Step 3: Frameworks

Step 3: Frameworks

Step 3: Frameworks

Step 3: Frameworks

Step 3: Frameworks

Step 3: Frameworks

tep 4: Profit

$

tep 4: Profit (utilize)

$

just one big pool of resources,utilize single machines more fully!

tep 4: Profit (utilize)

$

tep 4: Profit (utilize)

$

tep 4: Profit (utilize)

$

tep 4: Profit (utilize)

$

tep 4: Profit (utilize)

$

tep 4: Profit(statistical multiplexing)

$

tep 4: Profit(statistical multiplexing)

$

tep 4: Profit(statistical multiplexing)

$

tep 4: Profit(statistical multiplexing)

$

tep 4: Profit(statistical multiplexing)

$

tep 4: Profit(statistical multiplexing)

$

reduces CapEx and OpEx! tep 4: Profit(statistical multiplexing)

$

reduces latency! tep 4: Profit(statistical multiplexing)$

tep 4: Profit (failures)

$

tep 4: Profit (failures)

$

tep 4: Profit (failures)

$

This sounds pretty good!

Other than Hadoop, Spark, and Storm, what else can I run on Mesos? frameworksHadoop (github.com/mesos/hadoop)Spark (github.com/mesos/spark)DPark (github.com/douban/dpark)Storm (github.com/nathanmarz/storm)Chronos (github.com/airbnb/chronos)MPICH2 (in mesos git repository)Aurora (proposed for Apache incubator)

What about XYZ?port an existing frameworkstrategy: write a wrapper which launches existing components on mesos~100 lines of code to write a wrapper (the more lines, the more you can take advantage of elasticity or other mesos features)see src/examples/ in mesos repository

write a new framework!as a kernel, mesos provides a lot of primitives that make writing a new framework relatively easyprimitives: extracted commonality across existing distributed systems/frameworks (launching tasks, doing failure detection, etc) why re-implement them each time!? case study: chronosdistributed cron with dependenciesdeveloped at airbnb~3k lines of Scala!distributed, highly available, and fault tolerant without any network programming!http://github.com/airbnb/chronos

Hmm if Mesos gives me a datacenter computer can I run stuff other than analytics?case study: aurorarun N instances of my server, somewhere, forever(where server == arbitrary command line)developed at Twitterruns hundreds of production services, including ads!recently proposed for Apache Incubator!

aurora

aurora

aurora

aurora

aurora

But what about resource isolation!? I dont want my end users to have to wait for our website to load because of resource contention!resource isolationLinux control groups (cgroups)

CPU (upper and lower bounds)memorynetwork I/O (traffic controller)filesystem (lvm, in progress)

conclusionsdatacenter management is a pain

conclusionsmesos makes running frameworks on your datacenter easier as well as increasing utilization and performance while reducing CapEx and OpEx!

conclusionsrather than build your next distributed system from scratch, consider using mesosconclusionsyou can share your datacenter between analytics and online services!

Questions?mesos.apache.org@ApacheMesos

framework commonalityrun processes simultaneously (distributed)handle process failures (fault-tolerance)optimize execution (elasticity, scheduling)

primitivesscheduler distributed system master or coordinator(executor lower-level control of task execution, optional)requests/offers resource allocationstasks threads of the distributed system

schedulerApacheHadoop

Chronos

scheduler(1) brokers for resources(2) launches tasks(3) handles task terminationbrokering for resources(1) make resource requests 2 CPUs 1 GB RAM slave *(2) respond to resource offers 4 CPUs 4 GB RAM slave foo.bar.comoffers: non-blocking resource allocationexist to answer the question:what should mesos do if it cant satisfy a request?(1) wait until it can(2) offer the best allocation it can immediately

offers: non-blocking resource allocationexist to answer the question:what should mesos do if it cant satisfy a request?(1) wait until it can(2) offer the best allocation it can immediately

resource allocationApacheHadoop

Chronos

request

resource allocationApacheHadoop

Chronos

requestallocatordominant resource fairnessresource reservations

resource allocationApacheHadoop

Chronos

requestallocatordominant resource fairnessresource reservationsoptimisticpessimistic

resource allocationApacheHadoop

Chronos

requestallocatordominant resource fairnessresource reservationsoptimisticpessimisticno overlapping offersall overlapping offers

resource allocationApacheHadoop

Chronos

offerallocatordominant resource fairnessresource reservationstwo-level schedulingmesos: controls resource allocations to framework schedulersschedulers: make decisions about what to run given allocated resourcesend-to-end principleapplication-specific functions ought to reside in the end hosts of a network rather than intermediary nodestaskseither a concrete command line or an opaque description (which requires a framework executor to execute)

a consumer of resourcestask operationslaunching/killinghealth monitoring/reporting (failure detection)resource usage monitoring (statistics)

resource isolationcgroup per executor or task (if no executor)

resource controls adjusted dynamically as tasks come and go!case study: chronosdistributed cron with dependenciesbuilt at airbnb by @flo

before chronos

before chronossingle point of failure (and AWS was unreliable)resource starved (not scalable)

chronos requirementsfault tolerancedistributed (elastically take advantage of resources)retries (make sure a command eventually finishes)dependencies

chronosleverages the primitives of mesos~3k lines of scalahighly available (uses Mesos state)distributed / elasticno actual network programming!

after chronos

after chronos + hadoop

case study: aurorarun 200 of these, somewhere, foreverbuilt at Twitter

before aurorastatic partitioning of machines to serviceshardware outages caused site outagespuppet + monitops couldnt scale as fast as engineers

aurorahighly available (uses mesos replicated log)uses a python DSL to describe servicesleverages service discovery and proxying (see Twitter commons)

after aurorapower loss to 19 racks, no lost services!more than 400 engineers running serviceslargest cluster has >2500 machines

MesosMesosNodeNodeNodeNodeHadoopNodeNodeNodeNodeSparkNodeNodeMPIStorm

NodeChronosBefore and after pics139MesosMesosNodeNodeNodeNodeHadoopNodeNodeNodeNodeSparkNodeNodeMPINodeBefore and after pics140MesosMesosNodeNodeNodeNodeHadoopNodeNodeNodeNodeSparkNodeNodeMPIStormNodeBefore and after pics141MesosMesosNodeNodeNodeNodeHadoopNodeNodeNodeNodeSparkNodeNodeMPIStormNodeChronosBefore and after pics142Chart10.33333333330.33333333330.20000.33333333330.33333333330.33333333330.33333333330.33333333330

Series 1

Sheet1Series 1Series 20.0033%12100%1.0033%12100%2.0020%1260%3.000%210%4.000%280%5.000%0%6.0033%100%7.0033%100%8.0033%100%9.0033%100%10.0033%100%11.000%0%12.000%0%

Chart1000.33333333330.33333333330.33333333330.33333333330.33333333330.33333333330.33333333330.16666666