© 2014 MapR Technologies 1 Ted Dunning February 20, 2015.

© 2014 MapR Technologies 1© 2014 MapR Technologies

Project Myriad: Mesos and Yarn together

Ted Dunning

February 20, 2015

© 2014 MapR Technologies 2

Contact Information

Ted Dunning

Chief Applications Architect, MapR Technologies

Committer & PMC for Apache’s Drill, Zookeeper & Mahout

Mentor for Myriad & Apache’s Storm, Flink, Datafu, Optiq, Drill

Email [email protected] [email protected]

Twitter @ted_dunning @MapR@ApacheMyriad

Hashtag today: #StrataHadoop

mailto:[email protected]



Myriad Project

• Very new open source / open community project

• Started as collaboration between Mesosphere, MapR & eBay

• Proposal to be an incubator project of the Apache Foundation submitted 12 February 2015

• Goal: global resource management for multiple data centers


Agenda

• The need• Recap• How it works• Use Cases• Lessons Learned• The Future


What We Need

• Tight integration of resources and programming models• User specified resources and allocation models• Lightweight executive• Strong isolation• Fast task launch


What We Need

• Very fast scheduling• Very careful (slow) scheduling• Long-lived system tasks• Short-lived tasks• Long-lived ephemeral tasks• Pre-emption


What We Need

• Very good support of entire Hadoop eco-system– Tight integration of MapReduce2– Tez– Impala– Drill– Spark

• Very good support of everything else– Arbitrary containers– Web servers– Systems processes without containers– User defined containers– Licensing constraints


This is a problem


And an opportunity


What We Have - Yarn

• Resource Manager, NodeManager, heartbeat– Direct lineage from JobTracker, TaskTracker

• Application Master, Task containers– The other half of the JobTracker and TaskTracker

• Monolithic scheduling• Pre-emption• Hadoop standard• Pre-defined resources• Good Hadoop eco support

– MapReduce2, Tez, Impala, Drill, Spark


What We Have - Mesos

• Two level scheduling– Bottom level is application specific– Frameworks to ease complexity– Offers, Returns

• Actor-based, bidi RPC– Super fast process launch

• Marathon, Chronos– ISO8601, jboss, jetty, sinatra, rails

• User defined resources, attributes• Some Hadoop (Spark native!)


Sound the sameVery much not


Myriad integratesMesos and Yarn


How It Works

• Mesos creates virtual clusters

• YARN uses resources provided by Mesos

• Myriad can ask YARN to release some resources

• Or give it more

Mesos

YARN clusterYARNcluster

Web Servers


How Myriad Works

• Mesos runs Yarn– Yarn runs Yarn programs– Multiple Yarns supported– Multiple Yarn versions easy

• Mesos runs program + Yarn fakeout– Gets resources back from Yarn quickly– High priority “Yarn” program– As Yarn executes “tasks”, resources given back to Mesos– Allows fast spinup/spindown of Yarn resources


How Myriad Works

Mesos

Persistence Layer


Let’s see some examples


#1 – I wanna cluster


I Want a Cluster

• Very common need– Ephemeral clusters for multi-tenancy– Quick dev or QA clusters– Compatibility testing

• Yarn doesn’t run Yarn well– Especially across incompatible versions– Encapsulation can’t be unrolled

• Myriad does this trivially, but– Must have data localization, universal name space


#2 – Version upgrade


YARN Version Upgrade

• Another very common need– Need to test first– Applications roll over to new cluster– Resources follow applications– Data layer must remain inter-operable

• Yarn doesn’t run Yarn well (again)– Especially across incompatible versions– Encapsulation can’t be unrolled



#3 – Resource slosh


Resource Slosh

• Resource slosh– Data ingestion pulse requires many web-servers– After ingestion, analytics pulse requires many Hadoop nodes– Data layer must remain inter-operable

• Conflict between Sysop/Hadoop viewpoints



Some Lessons Learned

• Omega paper – Not news– Single scheduler framework not viable

• Multi-cultural software is actually pretty cool– But you have to value both cultures

• One incubator project (Slider) doesn’t change that


The Future

• Incubator– Proposal at http://wiki.apache.org/incubator/MyriadProposal– Initial team from Mesosphere, Ebay, MapR

• Community building– Diversity is good already– Starting with very lean team

http://wiki.apache.org/incubator/MyriadProposal



The Future

• Incubator– Proposal at http://wiki.apache.org/incubator/MyriadProposal– Initial team from Mesosphere, Ebay, MapR

• Community building– Diversity is good already– Starting with very lean team

• Older whisky, faster horses, more features– Apologies to the cowboy and the poet– And Tom T Hall



https://www.youtube.com/watch?v=vnvMcX95G20


World domination


World domination Peaceful coexistence via

specialization


Myriad Project

• Blog “Project Myriad: No Hadoop is an Island” http://bit.ly/myriad-mapr-blog

• Proposal to be an incubator project of the Apache Foundation submitted 12 February 2015 http://bit.ly/myriad-asf-proposal

• Initial code on github: http://bit.ly/github-myriad

• Join us! Twitter for Myriad community @ApacheMyriad[no, it’s not

an official project logo]

http://bit.ly/myriad-mapr-blog

http://bit.ly/myriad-asf-proposal

http://bit.ly/myriad-asf-proposal

http://bit.ly/github-myriad

http://bit.ly/github-myriad


Contact Information

Ted Dunning

Chief Applications Architect, MapR Technologies

Committer & PMC for Apache’s Drill, Zookeeper & Mahout

Mentor for Myriad & Apache’s Storm, Flink, Datafu, Optiq, Drill

Email [email protected] [email protected]

Twitter @ted_dunning @ApacheMyriad

Hashtag today: #StrataHadoop



© 2014 MapR Technologies 1 Ted Dunning February 20, 2015.

Documents

Transcript of © 2014 MapR Technologies 1 Ted Dunning February 20, 2015.