Container Orchestration Integration:

OpenStack Kuryr and

Apache Mesos@takufukushima

@takufukushima• MidoNet contributor sometimes

• OpenStack Kuryr contributor these days

Agenda1. Introduction to Docker and Apache

Mesos

2. The history of Docker and Apache Mesos Networking

3. OpenStack Kuryr as the building block

4. Summary

1. Introduction to Docker and Apache Mesos

We need the cluster manager• We distribute workloads to containers on

hosts or VM instances

• Docker and other containers are building blocks

• We want to manage them from the bird’s-eye view

Container cluster managers• Docker native

• Docker Swarm

• Apache Mesos

• Docker based

• Google Kubernetes

rocks

• Blazing fast (VM? Huh?)

• Great ecosystem

• e.g., DockerHub, Meetups

• Golang dev hipstersAnd nice art works

(seriously)

The dark side of

• “fundamentally flawed”

• “It’s The Future”

• “So I just need to split my simple CRUD app into 12 microservices, each with their own APIs which call each others’ APIs but handle failure resiliently, put them into Docker containers, launch a fleet of 8 machines which are Docker hosts running CoreOS, “orchestrate” them using a small Kubernetes cluster running etcd, figure out the “open questions” of networking and storage, and then I continuously deliver multiple redundant copies of each microservice to my fleet. Is that it?”

rocks• The core of Mesosphere DCOS

• Originally research project of UCB RAD (AMP) lab

• Great ecosystem and use cases

• Twitter, Apple, Airbnb, eBay and so on

• Pluggable frameworks

• Apache Aurora, Chronos, Marathon

architecture

Retrieved from http://radar.oreilly.com/2015/10/swarm-v-fleet-v-kubernetes-v-mesos.html

and

• Mesos has few containerizers

• cgroups and namespace based containerizer

• Docker containerizer

• External containerizer

2. The history of Docker and Apache Mesos Networking

Docker networking• docker0 bridge

• veth pairs and netns

• --icc and --link

• --net

• bridge, container, host, none

• NAT by iptables

Extended Docker networking• CoreOS flannel

• For Kubernetes

• Weave

• SocketPlane

• pipeworks

libnetwork• Networking component as a plugin

• docker network command

• Drivers separated from Docker core

• bridge

• overlay

• none

• Remote driver opened up for everyone

overlay driver• SocketPlane

• Container communication over the hosts

• VXLAN

• libkv for storing the network state in the distributed datastore

• --cluster-store and --cluster-advertise

• etcd, Consul and ZooKeeper

network.CreateEndpoint()

controller.NewNetwork()

InterfaceInfo

endpoint.Join() endpoint.Leave()

endpoint.Delete()

network.Delete()

sandbox.Info

e.g., netns

Network Controller

Driver

Built-in Remote

Endpoint

BridgeHostNone

OverlayCalicoKuryrWeave

Network Network

Endpoint Endpoint

IP addressesMAC addressesRoutesDNS entries

Container Container

OptionLabel

CLI

Sandbox Sandbox

Container Container

libnetwork CNM

networking

• Almost the same as Docker

• especially if you’re using Docker as the containerizer

• Containers share the IP of the slaves

• NAT and netns

integration point• External Containerizer Program (ECP)

• Slaves delegate the containerising to ECP

• It’s just building the Docker command

• Protobuf data is passed through stdin and stdout

• ENV vars can be used for additional data

new networking

• IPAM server

• IPAM client on masters and slaves

• Network Isolator Module (NIM) on slaves

• Cleanup Module on masters

new networking

Retrieved from https://github.com/apache/mesos/blob/master/docs/networking-for-mesos-managed-containers.md

3. OpenStack Kuryr as a building block

OpenStack? Why is it

related with Docker?

OpenStack and Docker• OpenStack and Docker are exclusive for each other at

this point

• Multi tenancy

• Strict resource isolation

• OpenStack Magnum

• Docker managed by OpenStack

• Docker containers on VM instances

• OpenStack Kolla

Revisiting OpenStack Neutron• Neutron is a networking component of OpenStack

• Networking resource allocation through the API

• Vendor agnostic APIs

• Many network controllers supporting these APIs

• The model of libnetwork is getting close to Neutron’s one

Maximizing the developers effects: Investment for the most effective way

OpenStack Kuryr• A new component in “Neutron Stadium”

• A translator between Neutron and libnetwork

• Map the API calls on the remote driver into Neutron’s API calls

OpenStack Kuryr• A new component in “Neutron Stadium”

• A translator between Neutron and libnetwork

• Map the API calls on the remote driver into Neutron’s API calls

OpenStack Kuryr• A new component in “Neutron Stadium”

• A translator between Neutron and libnetwork

• Map the API calls on the remote driver into Neutron’s API calls

Kuryr architcture

Neutron

Daemon

Host

DistributedDatastore

DistributedDatastoreDistributed

Datastore

ContainerContainerContainer

Daemon

Host

ContainerContainerContainer

Daemon

Host

ContainerContainerContainer

Kuryr Kuryr Kuryr

Keystone

Kuryr as a translator

Kuryr as a translator

4. Summary

Container networking made easy• Container networking had some issues

• The new networking models and APIs are emerging

• OpenStack Kuryr can be the common building block

Kuryr as a translator

The end of slides. Any questions?