JDD2014: Docker.io - versioned linux containers for JVM devops - Dominik Dorn

docker.ioversioned Lightweight Linux Containers

dominik dorn @ JDD 2014

2014-10-14 Dominik Dorn – docker.io @ JDD14

Overview

● Containers – Why? ● Containers vs. VMs● Docker● ( fig, jDocker, shipyard, Jenkins)● CoreOs● Panamax

Static website

Web frontend

User DB

Queue Analytics DB

Background workers

API endpoint

nginx 1.7.6 + modsecurity + openssl + bootstrap 2

postgresql + pgv8 + v8

hadoop + hive + thrift + OpenJDK

PlayFramework + sass

Redis + redis-sentinel

BatchEE, jRedis, libcurl + ffmpeg + libopencv + nodejs + phantomjs

Spray / Akka, jredis + postgresql-jdbc, anorm

Development VM

QA server

Public Cloud

Disaster recovery

Contributor’s laptop

Production Servers

The ChallengeM

ult

iplicit

y o

f S

tacks

Mu

ltip

licit

y o

f h

ard

ware

en

vir

on

men

ts

Production Cluster

Customer Data Center

Do s

er v

ices a

nd

ap

ps i n

tera

ct

ap

pro

pria

tely

?

Can

I mig

rate

sm

oot h

ly a

nd

q

uic

kl y

?

The Matrix From Hell

Static website

Web frontend

Background workers

User DB

Analytics DB

Queue

Development VM

QA Server

Single Prod

Server

Onsite Cluster

Public Cloud

Contributor’s

laptop

Customer Servers

? ? ? ? ? ? ?

? ? ? ? ? ? ?

? ? ? ? ? ? ?

? ? ? ? ? ? ?

? ? ? ? ? ? ?

? ? ? ? ? ? ?

Mu

ltip

licit

y o

f G

ood

s

Mu

ltip

ilic

ity o

f m

eth

od

s f

or

tran

sp

ort

ing

/sto

rin

g

Do I w

orry

ab

ou

t h

ow

good

s

inte

ract (e

.g.

cof

ee b

ean

s

next t o

sp

ices)

Can

I tran

sp

ort

qu

ickl y

an

d

sm

oot h

ly(e

.g. f ro

m b

oat to

tra

in t o

truck)

Cargo Transport Pre-1960

? ? ? ? ? ? ?

? ? ? ? ? ? ?

? ? ? ? ? ? ?

? ? ? ? ? ? ?

? ? ? ? ? ? ?

? ? ? ? ? ? ?

Also a matrix from hell

Mu

ltip

licit

y o

f G

ood

sM

ult

iplic

ity o

f m

eth

ods

for

tran

sport

ing

/sto

rin

gD

o I w

or ry

abou

t how

good

s inte

ract (e

.g.

coff

ee b

ean

s next to

sp

ices)

Can I tra

nsp

ort q

uickly

an

d sm

ooth

ly(e

.g. fro

m b

oat t o

train

to

truck)

Solution: Intermodal Shipping Container

…in between, can be loaded and unloaded, stacked, transported efficiently over long distances, and transferred from one mode of transport to another

A standard container that is loaded with virtually any goods, and stays sealed until it reaches final delivery.

Static website Web frontend User DB Queue Analytics DB

Development VM

QA server Public Cloud Contributor’s laptop

Docker is a shipping container system for code M

ult

iplic

ity o

f Sta

cks

Mult

iplic

ity o

f hard

ware

envir

onm

ents

Production Cluster

Customer Data Center

Do se

r vice

s and

apps in

tera

ct appro

pria

tely

?

Can I m

igra

te

smooth

ly a

nd

quickly

…that can be manipulated using standard operations and run consistently on virtually any hardware platform

An engine that enables any payload to be encapsulated as a lightweight, portable, self-sufficient container…

Static website

Web frontend

Background workers

User DB

Analytics DB

Queue

Development VM

QA Server

Single Prod

Server

Onsite Cluster

Public Cloud

Contributor’s

laptop

Customer Servers

Docker eliminates the matrix from Hell

AppA

Containers vs. VMs

Hypervisor (Type 2)

Host OS

Server

GuestOS

Bins/

Libs

AppA’

GuestOS

Bins/

Libs

AppB

GuestOS

Bins/

Libs

App

A

Docke

r

Host OS

Server

Bins/Libs

App A

Bins/Libs

App

B

App B

App

B

App B

VM

Container

Containers are isolated,but share OS and, whereappropriate, bins/libraries

GuestOS

GuestOS

…result is significantly faster deployment, much less overhead, easier migration, faster restart


Docker – Union File System

● aUFS – another UFS● GIT for file systems● Layering● Copy on Write FS● Can handle millions of

branches● Docker-Limitation (June

2014)● 127 Levels (w/o kernel mod)


Docker – installation

● On MacOS X 10.6 or newer



● On Ubuntu / Debian

wget -qO- https://get.docker.io/ | sh# does basically thissudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9$ sudo sh -c "echo deb https://get.docker.io/ubuntu docker main\> /etc/apt/sources.list.d/docker.list"$ sudo apt-get update$ sudo apt-get install lxc-docker



● If you would like to use Docker as a non-root user, you should now consider adding your user to the "docker" group with something like:

sudo usermod -aG docker domdorn

● domdorn@camelion ~ % docker –versionDocker version 1.0.0, build 63fe64c

● Use the install script, ubuntu/debian have old versions of docker in their Repos!


Docker – create a container

$ docker pull ubuntu$ docker run -i -t ubuntu /bin/bash// -i interactive// -t template*hackhack**hackhack*


Docker – commiting● domdorn@camelion ~ % docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS 3cf5b7a52e4d ubuntu:14.04 /bin/bash 9 minutes ago Up 9 minutes

● domdorn@camelion ~ % docker commit 3cf5b7a52e4d domdorn/nginx8b8401a1a847ec4ab9055e4d6a7698e5906727f8661a655abe8b614b7fdc9c61

● domdorn@camelion ~ % docker images REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZEdomdorn/nginx latest 8b8401a1a847 About a minute ago 349 MB


Docker – running● domdorn@camelion ~ % docker images

REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZEdomdorn/nginx latest 8b8401a1a847 About a minute ago 349 MB

● docker run -d -t domdorn/nginx /usr/sbin/nginx -g 'daemon off;'bdf571c2d6b7a9ef08b3d1e4795b0605c6034987478e1dfd4c185ba820c73435

● docker inspect --format '{{ .NetworkSettings.IPAddress }}' bdf571c2d6b172.17.0.9


Docker – publish ports● docker run -p 9000:80 -d -t domdorn/nginx /usr/sbin/nginx -g 'daemon off;'

d5efd9916468c6feed90021eaf862be3fdfd36cafa20b623e51efb1cebdb41ca


Docker – using volumes

● Map folders from the host / other containers into a container (bind-mount)docker run -v /var/www/www.jsug.at/htdocs:/usr/share/nginx/html --name 'jddhttp' -d -t domdorn/nginx /usr/sbin/nginx -g 'daemon off;' 29fcdbc6775d4200d14696299900fa88c1bc7da9e19a10866940eded69cd26a4

● docker inspect --format '{{ .NetworkSettings.IPAddress }}' jddhttp172.17.0.32


Docker – importing Volumes

● docker run --volumes-from=126995e77d9b -i -t ubuntu cat /usr/share/nginx/html/index.html <html><head><title>hello JDD</title></head><body><h1>Hello JDD!</h1></body></html>


Docker – linking containers● docker run --link=jddhttp:httpserver -i -t ubuntu ping -c 4 httpserverPING httpserver (172.17.0.25) 56(84) bytes of data.64 bytes from httpserver (172.17.0.25): icmp_seq=1 ttl=64 time=0.125 ms64 bytes from httpserver (172.17.0.25): icmp_seq=2 ttl=64 time=0.065 ms64 bytes from httpserver (172.17.0.25): icmp_seq=3 ttl=64 time=0.069 ms64 bytes from httpserver (172.17.0.25): icmp_seq=4 ttl=64 time=0.081 ms

--- httpserver ping statistics ---4 packets transmitted, 4 received, 0% packet loss, time 2999msrtt min/avg/max/mdev = 0.065/0.085/0.125/0.023 ms


Docker - Dockerfile

● Recipe to create Docker Containers● Commands

● ADD – copies files from the file system into the docker container● RUN – runs a command to build the image (e.g. apt-get install wget)● CMD – runs a command when the container gets instantiated ( e.g. apt-get update )● ENTRYPOINT – default command to run when starting a container (e.g. /usr/sbin/nginx )● ENV – set environment variables● EXPOSE – make a port available for incoming connections● FROM – base docker image, e.g. domdorn/nginx or “ubuntu”● MAINTAINER – name of the maintainer● USER – execute the entrypoint under a specific UID● VOLUME – defines directories that get exposed as volumes● WORKDIR – sets the workdir for the entrypoint


Fig

● http://orchardup.github.io/fig/index.html ● Define your app's environment with Docker so it

can be reproduced anywhere

Dockerfile:FROM orchardup/python:2.7ADD . /codeWORKDIR /codeRUN pip install -r requirements.txt

http://orchardup.github.io/fig/index.html


Fig

● Define the services that make up your app so they can be run together in an isolated environment:

fig.yml:web: build: . command: python app.py links: - db ports: - "8000:8000"db: image: orchardup/postgresql


Fig

● Then type fig up, and Fig will start and run your entire app:


jDocker

● https://github.com/nirima/jDocker● Fluent-API for Docker● Requires TCP-Socket enabled● Create/Delete/Start/Stop Docker Containers● Tests:

https://github.com/nirima/jDocker/blob/master/src/test/java/com/kpelykh/docker/client/test/DockerClientTest.java

https://github.com/nirima/jDocker





shipyard

● https://github.com/shipyard/shipyard● Webapp to manage docker containers● docker run -i -t -v

/var/run/docker.sock:/docker.sock shipyard/deploy setup

https://github.com/shipyard/shipyard


Shipyard - Management


Shipyard - Hosts


Shipyard - Metrics


jenkins

● https://wiki.jenkins-ci.org/display/JENKINS/Docker+Plugin

● Container/Slave per Build● Uses jDocker behind the scenes


Jenkins - Config


CoreOS

● Minimal & optimized OS to run Docker containers

● https://coreos.com/ ● “Linux for Massive Server Deployments

CoreOS enables warehouse-scale computing on top of a minimal, modern operating system.”

https://coreos.com/


CoreOS


Panamax

● By the Docker founders (Cloudcentric)● Apache 2 Licensed● Best practice combination of Docker-Tools

( fig, fleet, supervisord, coreos, etcd, …)● Web-UI for Docker-Configuration● Template-System● Panamax-Store: “AppStore” for Templates● Github integration


Links / Literature

● http://www.slideshare.net/dotCloud/docker-intro-november

● http://shipyard-project.com/ ● http://www.coreos.com ● http://orchardup.github.io/fig/index.html ● https://wiki.jenkins-ci.org/display/JENKINS/Doc

ker+Plugin● http://panamax.io

http://www.slideshare.net/dotCloud/docker-intro-november

http://www.slideshare.net/dotCloud/docker-intro-november

http://shipyard-project.com/

http://www.coreos.com/

http://orchardup.github.io/fig/index.html

https://wiki.jenkins-ci.org/display/JENKINS/Docker+Plugin

https://wiki.jenkins-ci.org/display/JENKINS/Docker+Plugin

http://panamax.io/


That's it!

twitter: @domdorn

mail: [email protected]

JDD2014: Docker.io - versioned linux containers for JVM devops - Dominik Dorn

Software

Transcript of JDD2014: Docker.io - versioned linux containers for JVM devops - Dominik Dorn