Download - Devoxx France 2015 - The Docker Orchestration Ecosystem on Azure

Patrick Chanezon, Docker Inc.@chanezon

The Docker Ecosystem

With slides from @jpetazzo @timpark @vieux @tnachen IBM

on Microsoft Azure

Ride the Whale!

French

Polyglot

Platforms

San Francisco

Developer Relations

@chanezon

1995 2015

Docker in the cloud market

Cloud Market

PublicHybridPrivate

IT Pros Devops DevelopersArchitects

History of containerization

• 1960’s mainframe

• 1990’s hardware virtualization

• 1990’s OS virt precursors: BSD Jails, Solaris zones

• 2006 Cloud IaaS

• 2009 platform virtualization (PaaS)

• 2013 Docker

See @bcantrill’s deck http://www.slideshare.net/bcantrill/docker-and-the-future-of-containers-in-production

http://www.slideshare.net/bcantrill/docker-and-the-future-of-containers-in-production

Happy birthday!

Why Docker success now?

• Cloud adoption

• Portability

• Hybrid

• Devops

It’s an ecosystem

Linux Container Ecosystem

Docker

Isolation using Linux kernel features

namespaces

pid

mnt

net

uts

ipc

user

cgroups

memory

cpu

blkio

devices

Docker language stacks

https://registry.hub.docker.com/_/java/

Docker now

A platform to build, ship, and run any app, anywhere

docker engine

docker hub

docker-machine

docker-compose

docker-swarm

Docker, the community

>700 contributors

~20 core maintainers

>40,000 Dockerized projects on GitHub

>60,000 repositories on Docker Hub

>25000 meetup members,>140 cities, >50 countries

>2,000,000 downloads of boot2docker

Docker Inc, the company

Headcount: ~130

Revenue: t-shirts and stickers featuring the cool blue whale

SAAS delivered through Docker Hub

Support & Training

soon: Docker Hub Enterprise, behind the firewall

It’s all about Devops

Separation of concerns:Dave the Developer

Inside my container: my code

my libraries

my package manager

my app

my data

Separation of concerns:Oscar the Ops guy

Outside the container: logging

remote access

network configuration

monitoring

Docker on Microsoft

Containers

Microsoft engaging with the Docker ecosystem

Windows Server Containers

Deploy almost anywhere

More Windows options

• Nano Server

• Hyper-V Containers

http://azure.microsoft.com/blog/2015/04/08/microsoft-unveils-new-container-technologies-for-the-next-generation-cloud

http://azure.microsoft.com/blog/2015/04/08/microsoft-unveils-new-container-technologies-for-the-next-generation-cloud

Docker on Azure

Azure Portal Ubuntu Docker VM

Azure x-plat CLI

VMNAME=jpetazzoIMAGE=b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04-LTS-amd64-server-20140724-en-us-30GBUSER=jpetazzoPASSWORD=1234abcdABCD@LOCATION="West US"

azure vm docker create $VMNAME \$IMAGE $USER $PASSWORD -l "$LOCATION"

export DOCKER_HOST=tcp://$VMNAME.cloudapp.net:4243docker --tls versionazure vm endpoint create $VMNAME 80

docker-machine

docker-machine create -d azure \

—azure-subscription-id="c4f51be3-784c-xxx-7c50ad9e1b7c" \

--azure-subscription-cert="/Users/pat/.ssh/docker-azure-

cert.pem" \

--azure-location="East US" \

--azure-size=Small \

--azure-username="pat" \

pat-docker-machine-n

Linux Container Ecosystem

Plugins

Flocker

Powerstrip

Prototyping Docker Plugins

https://clusterhq.com/blog/powerstrip-prototype-docker-extensions-today/

https://clusterhq.com/blog/powerstrip-prototype-docker-extensions-today/

Orchestration

Docker Swarm

Docker Swarm 0.2.0

@abronan - @aluzzardi - @vieux

Running containers on

multiple hosts

Today

Docker

CLI

Docker

CLIDocker

CLI

Introducing Docker Swarm

us-west us-east

Docker

CLIDocker

CLI

Swarm

Swarm in a nutshell

• Docker REST API (>85%)

• Resource management (CPU, Mem, Networking)

• Advanced scheduling with constraints and affinities

• Multiple Discovery Backends (hub, etcd, consul, zookeeper)

• TLS: Encryption & Authentication

TimelineOct

Nov

Dec

Jan

Feb

Jun

Proof of Concept

DockerCon EU

Open Repository

First Release Candidate

Swarm Beta Release

Global Hack Day

Open Proposal

Setup using the hosted discovery service

• Create a cluster:

$ swarm create

• Add nodes to a cluster:

$ swarm join --add=<node_ip> token://<token>

• Start Swarm

$ swarm manage --addr=<swarm_ip> token://<token>

Or you can use your own etcd, zookeeper or consul

Contributions are welcome :

Resource Management

• Memory

$ docker run -m 1g …

• CPU

$ docker run -c 1 …

• Ports

$ docker run -p 80:80 …

• More to come, ex: network interfaces

Constraints

• Standard constraints induced from docker info

docker run -e “constraint:operatingsystem==*fedora*” …

docker run -e “constraint:storagedriver==*aufs*” …

• Custom constraints with host labels

docker -d --label “region==us-east”

docker run -e “constraint:region==us-east” …

• Pin a container to a specific host

docker run –e “constraint:node==ubuntu-2” …

Affinities

• Containers affinities

docker run --name web nginx

docker run -e “affinity:container==web” logger

• Containers Anti-affinities

docker run --name redis-master redis

docker run --name redis-slave -e “affinity:container!=redis*”

…

• Images affinities

docker run -e “affinity:image==redis” redis

New in 0.2.0: Soft Affinities/Constraints

• Containers affinities

docker run -e “affinity:container~!=—name web nginx

docker run -e “affinity:container==web” logger

• Containers Anti-affinities

docker run --name redis-master redis

docker run --name redis-slave -e “affinity:container!=redis*”

…

• Images affinities

docker run -e “affinity:image==redis” redis

Swarm Scheduler

2 steps:

• 1- Apply filters to exclude nodes

- ports

- labels

- health

• 2- Use a strategy to pick the best node

- random

- binpack

- spread

Contributions are welcome :

Swarm Beta: Integrations

• Fully integrated with Machine

• Partially integrated with Compose

• Mesos integration has started in collaboration with Mesosphere.

CoreOS

Docker & etcd

Cluster Architecture

https://coreos.com/docs/cluster-management/setup/cluster-architectures/

CoreOS / Docker / Spring Boot

https://github.com/chanezon/azure-linux/tree/master/coreos/cloud-init

Deis (http://deis.io)

• Open source PaaS platform that builds on CoreOS.• Replicates the popular Heroku devops workflow.

• Primary mechanism for pushing applications is through git.• Developer experience is not unlike Azure Websites…• …but is built on Linux so full support for open source stacks.

• Enables us to win migrations from Salesforce to Azure.• Hackfest in November to enable Deis for Tagboard.

• Enables us to win startups that expect this workflow.

tpark:www$ git push deis master

• Git pushes master to deis git remote on endpoint• Deis senses static web application• Selects Heroku Buildpack• Uses buildpack to build application Docker container.• Pushes this container to a private Docker registry.• Orchestrates the creation or update of this container

on the cluster.• Updates routing mesh to route to these containers.

Router Mesh

deis-1 deis-2 deis-3 deis-4

www

CoreOS CoreOS CoreOS CoreOS

tpark:www$ deis scale www=3

• Deis pushes the container to two more cluster nodes.• Updates routing mesh to pass traffic to these nodes.

Router Mesh


www www www

tpark:api$ git push deis master

• Git pushes master to deis git remote on endpoint• Deis senses node.js application• Selects Heroku node.js Buildpack• Uses buildpack to build application Docker container.• Pushes this container to a private Docker registry.• Orchestrates the creation or update of this container

on the cluster.• Updates routing mesh to route to these containers.

Router Mesh


www

api

www

api

www api

Router Mesh


www

api

www

api

www

api

tpark:api$ deis config:set DATABASE_URL=postgres://user:[email protected]:5432/db

• Applications in Deis are configured through environmental variables.

• MUST READ: http://12factor.net/• Key point: Code is separated from config. • Enables generic containers that are configured at runtime.• Every app container spun up by Deis will have a copy of these

config environmental variables.

http://12factor.net/

tpark:api$ deis logs

• Deis automatically rolls and consolidates logs from all containers.

Router Mesh


www

api

www

api

www

api

Kubernetes

Kubernetes (http://kubernetes.io)

KubernetesMaster / Scheduler

host-1 host-2 host-3 host-n

…..Container Agent Container Agent Container Agent Container Agent

Linux Linux Linux Linux

KubernetesScheduler


…..Container Agent Container Agent Container Agent Container Agent

Linux Linux Linux Linux

Container

Container

Kubernetes

host-1

Container


…

Container

Container

Container

Container

ContainerContainer

ContainerContainer

Kubernetes

host-1 host-2 host-3 host-4 host-n

…

Frontend

Worker

my_app pod

MyAppMyApp MyApp

Replication Controller

3

Kubernetes


…MyAppMyApp MyApp


Pod Pod

Pod

Pod

PodPod

PodPod


Kubernetes


…MyAppstaging

MyAppstaging

MyAppstaging

MyAppprod

MyAppprod

MyAppprod

MyAppprod

MyAppprod

MyApp Production Service{ environment: prod }

MyApp Staging Service{ environment: staging }

Labels and Services

Cloud Foundry & IBM BlueMix

Cloud Foundry Diego & Lattice

cf docker-push my-app cloudfoundry/lattice-app

IBM BluemixThe Digital Innovation Platform

97

Customer Managed

Service Provider Managed

IBM SoftLayer

Bluemix started as a public PaaSBluemix started with a major focus on developer productivity in the public cloud.

Infrastructure as

a Service

Code

Data

Runtime

Middleware

OS

Virtualization

Servers

Storage

Networking

Code

Data

Runtime

Middleware

OS

Virtualization

Servers

Storage

Networking

Platform as

a Service

98

Customer Managed

Service Provider Managed

IBM SoftLayer

We listened. Now we’re evolving to become even more flexible.

Capabilities in Bluemix now span PaaS and IaaS and can be delivered as a public,

dedicated, or on-premises* implementation.

Infrastructure as

a Service

Code

Data

Runtime

Middleware

OS

Virtualization

Servers

Storage

Networking

Code

Data

Runtime

Middleware

OS

Virtualization

Servers

Storage

Networking

Platform as

a Service

*Bluemix Local coming Summer 2015

Built on open

technologies:

How does Bluemix work?Bluemix is underlined by three key open compute technologies: Cloud Foundry, Docker, and

OpenStack. It extends each of these with a growing number of services, robust DevOps tooling,

integration capabilities, and a seamless developer experience.

99

Flexible Compute Options to Run Apps / Services

Instant Runtimes Containers Virtual Machines

Platform Deployment Options that Meet Your Workload Requirements

Bluemix

Public

Bluemix

Dedicated

Bluemix

Local*

DevOps

Tooling Your Own Hosted Apps / Services

Integration and

API Mgmt

Powered by IBM SoftLayer In Your Data Center

+ + +

+ +

+ Always focused on what’s next

Catalog of Services that Extend Apps’ Functionality

Web Data Mobile AnalyticsCognitive IoT Security Yours

+

*Bluemix Local coming Summer 2015

Containers in BluemixBluemix now comes with a fully integrated, high performance Docker experience, meaning monitoring,

logging, elasticity, enterprise images, and VM abstraction are all standard.

100

Docker Value IBM Value-add Customer Value

Docker Hub Registry holds a

repository of 75000+ Docker

images

• IBM hosted public registry containing IBM images - linked to

Docker Hub

• Client unique registry available on and off premises

• Enterprise-ready images

Access to the images you require to deploy

containers that meet your business needs and

strategy

Open-source, standardized,

lightweight, self sufficient LXC

container technology

• Enhanced performance with bare metal deployment

• Run images to local datacenter or cloud

• Deployment choice with pSeries & zSeries

Flexibility to choose the right hybrid cloud mix for your business

Build, ship, and run standardized

containers

• Integrated monitoring & logging

• Elasticity to grow storage & container needs

• Life-cycle management of containers and data volumes

• No VMs to manage

Docker ease of use combined with enterprise-

level integrity and confidence

Container connections using

links and service discovery

• Private network communication

• External IP address

• Subnet Range

Extends and connects Docker containers to

production-ready enterprise environments

Others

Joyent TritonThe network is the computer… v2:-)

CleverCloud

RancherOS

Orchestration summary• Docker Swarm: Docker-style, provision with docker-machine

• Mesos: Twitter-style, aligned with Swarm

• Fleet: CoreOS-style, simple

• Kubernetes: Google-style, heavy-duty, many concepts

• Deis: Heroku-style workflow

• Cloud Foundry Diego, IBM BlueMix: PaaS -> orchestration

• Also: Joyent, Tutum, Flynn

Fire up your first container today!

on Microsoft Azure

Ride the Whale!

Learning

• https://github.com/chanezon/azure-linux

• Docker container to get started

docker run –ti chanezon/linux

• CoreOS cluster, fleet

• Deis

• Weave

• docker-machine

• Deploy Java app

https://github.com/chanezon/azure-linux

10

3

References• talk about cloud platforms: Managing complexity in giant systems http://www.slideshare.net/chanezon/tackling-

complexity-in-giant-systems-approaches-from-several-cloud-providers

• talk about Devops, the Microsoft Wayhttp://www.slideshare.net/chanezon/devops-the-microsoft-way

• MS Open Tech https://msopentech.com/ Blog, VM Depot

• P@ Linux on Azure pages https://github.com/chanezon/azure-linux/

• Tim’s CoreOS tutorial https://github.com/timfpark/coreos-azure

• Tim’s Deis documentation

• @jpetazzo’s presentations http://www.slideshare.net/jpetazzo/

• @bcantrill’s deck http://www.slideshare.net/bcantrill/docker-and-the-future-of-containers-in-production

• @vieux deck on Swarm

• @htchen deck on Mesos + Swarm https://speakerdeck.com/tnachen/docker-swarm-plus-mesos

http://www.slideshare.net/chanezon/tackling-complexity-in-giant-systems-approaches-from-several-cloud-providers

https://msopentech.com/

https://github.com/timfpark/coreos-azure

http://www.slideshare.net/jpetazzo/

http://www.slideshare.net/bcantrill/docker-and-the-future-of-containers-in-production