Container cluster management solutions

Kishore JagannathSystem ArchitectHewlett Packard Enterprise

Mesos/Marathon Architecture• Apache Mesos is a resource manager and provides a

datacenter view of a cluster of machines.• Works in a Master/Slave architecture.• Master schedules job on slaves which is executed by an

executor.• Masters are connected to service registry such as Zookeeper

for leader election and HA.• Offers resources to any framework connected on top of it.• Marathon is a scheduler/framework providing orchestration

capabilities.

Mesos MasterAssigns jobs to slaves and make resource offers to frameworkMesos SlaveRuns Mesos agents. Executes tasks and provides resource capacity to masterZookeeperLeader election of MastersFrameworkSelects and schedules tasks via resource offers. Deploys executors on agents to execute scheduled tasks

Mesos/Marathon Overview Demo

Docker Swarm Architecture docker swarm init <<options>>docker swarm join <<options>>

• Directly available within Docker engine from docker 1.12.• Enable cluster setup in swarm mode with simple commands• Create services instead of containers. • Managers talk to workers to schedule tasks on the worker.• Managers use Raft algorithm for leader election.• Worker nodes communicate through gossip protocol.• In built support for Service Registry and discovery. • No need for external services like consul, etcd.• In built Load balancing.• Multi host container networking via Overlay networks• Secured by default via TLS.• Automatic reconciliation to desired state of cluster.

MasterSchedules tasks on slaves and exposes service commands.SlaveExecutes containersInternal Key Value StoreInbuilt key values store for master leader election using Raft algorithm

Docker Swarm Overview Demo

Kubernetes Overview

• APIServerInteraction point with Kubernetes• SchedulerSchedules the jobs on slaves • Controller ManagerEnsures the desired state of pod replicas• ETCDKey value store enabling master election• KubeletAgent running on slave nodes• Kube ProxyA small component in slave nodes enabling service discovery across Pods

A group of tightly dependent containers within a POD and exposed as a Service. Allows specifying the desired replicas of pods within a service. Inbuilt support for service registry and load balancingAutomatic reconciliation to desired state

Kubernetes Overview Demo

Comparison TableMesos/Marathon Swarm Kubernetes

Background Open Source based on Apache Mesos and productized by Mesosphere DCOS

Open Source. Docker Community

Open Source. Google

Workloads Multiple workloads Container based workloads

Container based workloads

Desired state Supported by Marathon framework

Supported by service replicas

Supported via replication controllers

Unit of scalability Container Container Pods

High availability Application containers distributed on Mesos slaves

Application containers distributed on swarm workers

Pods distributed among slave nodes

Load Balancing load balanced through Mesos DNS or Marathon LB

Container tasks abstracted as a service and requests are load balanced.

Pods exposed as services and requests are loadbalanced

Mesos/Marathon Swarm Kubernetes

Service discovery Dns entries in case of mesos-dns or service ports in marathon-lb(HA Proxy) in each node

Services discovered via service names enabled via gossip protocol in target nodes

Pods talk to each other via Service names enabled through kube-proxy running in nodes.

Key value stores ZooKeeper Inbuilt etcd

Rolling update support

Yes. Can specify minimal health capacity

Yes. Can specify time interval between updates

Yes. Can specify time interval between updates.

Networking Expose container ports to host ports to expose. Service ports used for managing cluster

Container ports not exposed. Service ports can be published. Inbuilt Overlay network

Containers within pod share network space. Across pods clusterIp and external through serviceport. UsesFlannel for overlay network

Support for stateful applications

Yes with Datastax enterprise and RexRay

No Yes from kubernetes 1.3. concept of PetSet

Salient FeaturesMesos/Marathon• Timetested: Apache Mesos as a resource manager has been around long time even

before containers• Support for containerized as well as non containerized workloads• Easy to configure health check of applications apart from nodes.• Open architecture. Support for multiple frameworks and multiple schedulers.

Docker Swarm• Have been around for more than a year but given new life with in built swarm mode

and docker services from 1.12• Available inbuilt with docker engine so no additional installation. Super simple to

configure just two commands.• Support for service registry store, overlay networks and load balancing is inbuilt.

Salient Features(contd.)• Will be soon integrated with other Docker solutions such as UCP in future releases.• Exposed as Docker cli commands, very easy to learn and adapt for folks familiar with

Docker.

Kubernetes• One of the initial popular clustering solutions around containers.• Provides the concept of pods enabling closely dependent containers to communicate

in same network space via localhost.• Pods, Services, Deployments are discrete pieces and glued through selectors

providing more flexibility.• Inbuilt service registry and load balancing.