Musings on Mesos: Docker, Kubernetes, and Beyond

Click here to load reader

  • date post

  • Category


  • view

  • download


Embed Size (px)


2014 ApacheCon EU talk on Mesos and it's integration with Docker as well the ecosystem, and outline the future of Kubernetes integration

Transcript of Musings on Mesos: Docker, Kubernetes, and Beyond

  • 1. Musings on Mesos:Docker, Kubernetes, and BeyondTimothy St. Clair@timothyscMesos committer and PMC member11/19/2014INTERNAL ONLY

2. Audience Poll - Mesos I have ...1. Only heard talks about Mesos, nothing more.2. Played with Mesos on a sample cluster, initial POC.3. Have a Mesos cluster in a production environment4. DUDE.!.! I wrote a custom Mesos framework inHaskell before breakfast Bro ...INTERNAL ONLY 3. Quick Overview Mesos 101 CondensedAt its core, Mesos is a focused, scalable, two-phasedmeta-scheduler that provides primitivesto express a wide variety of scheduling patternsand use cases. Solutions are written atop ofMesos as frameworks. Frameworks are targetedfor a particular use case, and maintain domainspecific information. By remaining focused at itscore, Mesos is not architecturally encumbered bydomain specific problems that often exist in othermonolithic schedulers.INTERNAL ONLY 4. The Response I Always Hope ForINTERNAL ONLY 5. Typical ResponseINTERNAL ONLY 6. 101 Unpacking: Meta-Scheduling?At its core, Mesos is a focused, scalable, meta-schedulerthat provides primitives to express awide variety of scheduling patterns and usecases. It means that Mesos enables other distributedapplications to define their own schedulingpolicies, with an core algorithm (DRF) to shareresources across those applications. Mesos can run on 1-O(10^4) nodes.So, it is a scheduler... that allows distributedapplications to share resources in a cluster.INTERNAL ONLY 7. 101 Unpacking: frameworks?Solutions are written atop of Mesos asframeworks, and are targeted for a particular usecase. Framework = distributed application =scheduler Framework could be anything from batch,application, or micro-service scheduling.INTERNAL ONLY 8. 101 Unpacking: Architecturally Encumbered?Mesos is not architecturally encumbered bydomain specific problems that often exist withinother monolithic schedulers. Mesos does not subsume domain specificproblems, that is a frameworks job.INTERNAL ONLY Service scheduling Fault tolerance Load balancing Discovery Batch Scheduling Quotas (user/group) Limits 9. 101 Motivation: New Reality New applications need to be: Fault tolerant (Withstand failure) Scalable (Not crumble under it's own weight) Elastic (Can grow and shrink based on demand) Multi-tenent (It can't have it's own dedicated cluster) Must play nice with the other kids. So what does that really mean?INTERNAL ONLY 10. 101 Motivation: Distributed Application There's Just No Getting Around It: You're Building a Distributed System MarkINTERNAL ONLYCavage Key takeaways on architecture: Decompose the business applications into discrete services on the boundaries offault domains, scaling, and data workload. Make as many things as possible stateless When dealing with state, deeply understand CAP, latency, throughput, anddurability requirements.Without practical experience working on successfuland failedsystems, mostengineers take a "hopefully it works" approach and attempt to string together off-the-shelfsoftware, whether open source or commercial, and often are unsuccessful at building aresilient, performant system. In reality, building a distributed system requires a methodicalapproach to requirements along the boundaries of failure domains, latency, throughput,durability, consistency, and desired SLAs for the business application at all aspects of theapplication. 11. 101 Summary : Distributed Systems KernelCluster Manager (google-ism) Mesos is built using the same principles as the Linuxkernel, only at a different level of abstraction.We wanted people to be able to program for the data-centerjust like they program for their laptop ~ BenINTERNAL ONLYHindman Computer : Data-center Kernel : Mesos Application : Distributed application Operating System : (Mesos+Frameworks+Ecosystem) 12. So Having a Kernel for Distributed SystemsMakes a Lot of Sense.?.? _()_/INTERNAL ONLY 13. Mesos @ 50K Hadoop Spark StormINTERNAL ONLYWorkloadsKernelbatch servicesFrameworksMarathon Kubernetes Auroradistributed file system DFSdistributed resources: CPU, RAM, I/O, FS, rack locality, etc. Cluster 14. Audience Poll - Docker I have ...1. Only heard talks/news about Docker.2. Played with Docker and use it on a single host.3. Have Docker running in a production cluster.4.DUDE.!.! I've ran 300 Million containers last month.INTERNAL ONLY 15. Docker Docker DockerCombination of technologies cgroups + namespaces + image format + lifecyclemanagement + application virtualizationGit like semantics around image buildingINTERNAL ONLY pull, commit, push ... Application virtualization Native Speeds >> Density the virtual machines 16. The 'Holy Grail' of Clustering Provide maximum application density per-machine(currently in the hundreds).INTERNAL ONLY >> Then virtualization Improved 'application namespace' isolationacross your cluster No longer need to ensure stacks | sub-stacks arerolled onto your cluster. Enables multiple versions across a cluster easilyCombined with clustering, it can obviatetraditional 'deployment problems' 17. Typical ResponseINTERNAL ONLY 18. RealityDocker 90% of the things,but that last 10% of thingsmay require super-privilegedINTERNAL ONLYcontainers,where the container hasfull access to the systemand that gets kind offunky because then youcould have dependencyleakage... and potentiallya host of other versioningissues. And uhhhhhhh.. 19. Reality The last 10% Applications that want orneed to load customkernel drivers. Low level systemapplications(namespaces) Cluster managers, suchas Mesos whichleverage namespaces.INTERNAL ONLY 20. Reality Networking is thorny, withmany solutions O(10^2) Pipework Flannel Weave vSwitch What happens whendensity is on the order1000's. What happens if youwanted to have a densityon the order of 10,000+.INTERNAL ONLY 21. 10,000+ .?.? What is the averageutilization of your datacenter? Twitter & Google runbetween 20-30%(Quasar paper) 640K ought to beenough for anybody. -B.Gates.INTERNAL ONLY 22. WhateverINTERNAL ONLY 23. (Apache Infra) What if.... There Were OfficialApache Repositories on Docker-Hub.INTERNAL ONLY 24. Mesos and DockerIntegration"If a Docker application is a Lego brick, Kubernetes would belike a kit for building the Millennium Falcon and the Mesoscluster would be like a whole Star Wars universe made ofLegos." ~ SolomonINTERNAL ONLY 1st classed in 0.20 A lot of feedback and updated 0.21 (soon-ish) Service Scheduling Marathon + HA-Proxy + Docker 10^4 Kubernetes (WIP) Batch Spark ( Chronos 25. Mesos and DockerUse Cases Elastic applications in a shared multi-tenant environmentRAILS Memcached Spark100%75%50%25%Graphic Courtesy of Paco NathanINTERNAL ONLY100%75%50%25%0%100%75%50%25%0%0%tt100%75%50%25%0% 26. Can I Write My Own Framework That UsesDocker?INTERNAL ONLY YES! Mesos has 1st class Docker support, by providing an APIDockerInfo {} BUT... Writing a framework in development is relatively easy Writing an application that is highly available, scaleswell, can be difficult. HOPE Creation of higher order libraries (libc) to help make iteasier to create frameworks. 27. Can you run Mesos from a Docker Container? Current State sort of. mesos-master Using bridged networkingyou can easily run a HAconfiguration mesos-slave mesos requires low levelsystem access whichmakes it difficult to put ina container and maintainfull feature parity TBD with superprivileged containersINTERNAL ONLY 28. How are we doing? _()_/INTERNAL ONLY 29. Audience Poll - Kubernetes I have ...1. Only heard talks/news about kubernetes.2. Played with kubernetes and use it on a single host.3. Have kubernetes running on GCE.INTERNAL ONLY 30. Kubernetes (WIP) Kubernetes is a system for managing containerizedapplications across multiple hosts, providing basicmechanisms for deployment, maintenance, and scalingof applications. Its APIs are intended to serve as thefoundation for an open ecosystem of tools, automationsystems, and higher-level API layers. Kubernetes establishes robust declarative primitives formaintaining the desired state requested by the user.These primitives are the main value added byKubernetes.INTERNAL ONLY pre-production beta. 31. Kubernetes Pods are the atom of scheduling, and are agroup of containers that are scheduled onto thesame host. Pods facilitate data sharing and communication.INTERNAL ONLY Shared mount point Shared network namespace/IP and port space Higher order abstraction to the low level interface ofcontainers Composable micro-services 32. Kubernetes pod.jsonINTERNAL ONLY{ "apiVersion": "v1beta1","kind": "Pod","id": "redis-master-pod","desiredState": {"manifest": {"version": "v1beta1","id": "redis-master-pod","containers": [{"name": "redis-master","image": "gurpartap/redis","ports": [{ "name": "redis-server", "containerPort": 6379 }] ... + labels 33. Kubernetes frameworkfor Mesos (WIP) Kubernetes enables the Pod (group of co-located containers)abstraction, along with Pod labels for service discovery, load-balancing,and replication control. Mesos provides the fine-grainedresource allocations for pods across nodes in a cluster,and can make Kubernetes play nicely with other frameworksrunning on the same cluster resources. Provide advanced scheduling capabilities (grouping,spreading ...)INTERNAL ONLY Availability zone fail-over Elasticity Scale up and down kublets MORE TO COME... 34. Kubernetes vs. Mesos?INTERNAL ONLY Not at all! Kubernetes is an opinionated declarative model on how toaddress micro-services Mesos provides an imperative framework by whichapplication developers can define scheduling policy in aprogrammatic fashion. When leveraged together it provides a data-center with theability to both. 35. Kubernetes - Mesos = StaticProvisioningSpark KubernetesINTERNAL ONLY 36. Hows your social media updates?#Mesos + #ApacheCon