Resource Sharing Beyond Boundaries - Apache Myriad
-
Upload
santosh-marella -
Category
Technology
-
view
322 -
download
0
Transcript of Resource Sharing Beyond Boundaries - Apache Myriad
![Page 1: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/1.jpg)
Resource Sharing Beyond Boundaries Mohit Soni Santosh Marella
![Page 2: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/2.jpg)
![Page 3: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/3.jpg)
![Page 4: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/4.jpg)
Adam Bordelon Anoop Dawar Ben HindmanBrandon Gulla Danese Cooper Darin JohnsonJim Klucar Kannan Rajah Ken SipeLuciano Resende Meghdoot Bhattacharya Paul ReedRenan DelValle Ruth Harris Shingo OmuraSwapnil Daingade Ted Dunning Will OchandarenaYuliya Feldman Zhongyue Luo
![Page 5: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/5.jpg)
AgendaWhat's up with Datacenters these days?Apache Mesos vs. Apache Hadoop/YARN?Why would you want/need both?Resource Sharing with Apache Myriad
![Page 6: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/6.jpg)
What's running on your datacenter?Tier 1 servicesTier 2 servicesHigh Priority BatchBest Effort, backfill
![Page 7: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/7.jpg)
RequirementsProgramming models based on resources, not machinesCustom resource typesCustom scheduling algorithms:Fast vs. careful/slowLightweight executors, fast task launch timeMulti-tenancy, utilization, strong isolation
![Page 8: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/8.jpg)
Hadoop and MoreSupport Hadoop/BigData ecosystemSupport arbitrary (legacy) processes/containersConnect Big Data to non-Hadoop apps, share data, resources
![Page 9: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/9.jpg)
Mesos from 10,000 feetOpen Source Apache project Cluster Resource Manager Scalable to 10,000s of nodes Fault-tolerant, no SPOF Multi-tenancy, Resource Isolation Improved resource utilization
![Page 10: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/10.jpg)
Mesos is more thanYet Another Resource Negotiator
Long-running services; real-time jobs Native Docker; cgroups for years;
Isolate cpu/mem/disk/net/other Distributed systems SDK;
~200 loc for a new app Core written in C++ for performance,
Apps in any language
![Page 11: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/11.jpg)
![Page 12: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/12.jpg)
![Page 13: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/13.jpg)
![Page 14: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/14.jpg)
Why two resource managers?
![Page 15: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/15.jpg)
![Page 16: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/16.jpg)
Static Partitioning sucksHadoop teams fine with isolated clusters, but Ops team unhappy; slow to provisionResource silos, no elasticityWant to run Hadoop on the same infrastructure, without interrupting Tier-1 servicesWant multi-tenancy, resource sharing/isolation
![Page 17: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/17.jpg)
![Page 18: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/18.jpg)
Introducing Myriad
![Page 19: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/19.jpg)
Myriad OverviewMesos Framework for Apache YARNMesos manages DC, YARN manages HadoopCoarse and fine grained resource sharing
![Page 20: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/20.jpg)
Resource Sharing
![Page 21: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/21.jpg)
![Page 22: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/22.jpg)
![Page 23: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/23.jpg)
![Page 24: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/24.jpg)
![Page 25: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/25.jpg)
![Page 26: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/26.jpg)
![Page 27: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/27.jpg)
![Page 28: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/28.jpg)
![Page 29: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/29.jpg)
![Page 30: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/30.jpg)
![Page 31: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/31.jpg)
![Page 32: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/32.jpg)
![Page 33: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/33.jpg)
![Page 34: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/34.jpg)
![Page 35: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/35.jpg)
![Page 36: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/36.jpg)
![Page 37: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/37.jpg)
![Page 38: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/38.jpg)
![Page 39: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/39.jpg)
Demo
![Page 40: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/40.jpg)
Myriad improves Mesos Tighter integration with Hadoop frameworks
like HBase, Hive, PigBorrow resources from Hadoop when traffic spikes for tier-1 servicesBackfill unused resource capacity with best-effort Hadoop jobs
No Mesos code changes necessary
![Page 41: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/41.jpg)
Myriad improves Hadoop Elastic scaling Fault-tolerant: Maintain NM capacity Share resources with other workloads,
improve resource utilization High SLA hadoop jobs unaffected No YARN/Hadoop code changes
![Page 42: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/42.jpg)
Other FeaturesRM failover/discovery using Marathon/Mesos-DNSDistribution of hadoop binariesWeb InterfaceMyriad scheduler HA, task reconciliation(in progress)Ability to launch Job History Server(in progress)Your favorite feature here!
![Page 43: Resource Sharing Beyond Boundaries - Apache Myriad](https://reader031.fdocuments.net/reader031/viewer/2022021918/58a7c3c11a28ab6b5a8b52e7/html5/thumbnails/43.jpg)
Learn More!https://github.com/mesos/myriad
[email protected] JIRA
Apache Myriad Incubator ProposalApache Myriad Incubator Status Page