more DEVs on DEVOPS withDOCKER ORCHESTRATIONAll the Things you have to Know in Order to use docker with Orchestration in Production like I did but not really like i did it because I cried, I really cried hard, so please do not cry because life is short and you should spend your time enjoying it.

This talk is for developers that know what docker is,And maybe they already use in dev,Or in test env.

How many of you is using dokcer in dev?And in production?

This would be a gentle introduction to Docker

-----------

6 Fino ll1 exlFino al devops 12min

2 .@liuggio Giulio De Donato

@liuggio

welcometothebundle.com

more DEVs on DEVOPS withDOCKER ORCHESTRATIONAll the Things you have to Know in Order to use docker with Orchestration in Production like I did but not really like i did it because I cried, I really cried hard, so please do not cry because life is short and you should spend your time enjoying it.

It is 2016 and Docker is everywhere, in the workshop, in the conferences, startup sites, including Amazon and services in google app engine with kubernetes, got billion on big fund, all the big companies are moving to this trend.Can you read the real titleThis talk is about all the thinks I think you have to know in order to start working in production with docker...A lot things to know

Did you read the title?Did you carefully read the title?

The docker ecosystem is very loudy, this has lead to much hype, and some disillusion, around this space.

We are here to cut through some of this confusion, and explains how containers are actually being used within the enterprise.

Is organized in things to know

@liuggio Giulio De Donato

1Docker Container!==

The docker ecosystem is very loudy, this has lead to much hype, and some disillusion, around this space.We are here to cut through some of this confusion, and explains how containers are actually being used within the enterprise.

The developer worked hard to integrate

Things to know n1. Docker wasnt a new technology...

But the container story start really a lot of years ago Since 2002 this technology has made steps forward, thanks to different sponsors (IBM, Ubuntu, Google etc)

Un po di storia...API!TODO storia

So docker didnt invent the container technologyThey added great marketing, great api, they added values and simplify the workflow for a developer

Docker is a company is an open source organization that pushed forward allof this they used golang :)----------I held a workshop about docker and an attendee told me oh yes Docker i know the virtualization thing

------5min to devops

@liuggio Giulio De Donato

Virtualization !== Container

Container are mostly syscalls2

Days ago when I talked to my friend about the workshop on docker I held, he said "ah the virtualization think?" i answered yes but this association between virtualization and containers is conceptually similar but technically so far,

Prima di arrivare alla ragione avete mai visto questa immagine,

ovunque in tutti i talk si vede questa immagine, semplice per simulare un isolamento non serve creare un altro systema operativo sopra al systema operativo esistente, eh no,Basta usare la tecnologia dei container che non sono altro che patch e syscall,Quindi non virtualizzazione sono chiamate a sistema, lisolamento il problema.

In 2012 Glauber Costa tells at the LinuxCon Europe: "I once heard That hypervisors are the living proof of operating system's incompetence" - Glauber Costa's - LinuxCon Europe 2012

Dont you think that buying any hardware or create a sub-operating system (VM) only to defend against a fork bomb is a little too much?

@liuggio Giulio De Donato

3It is all about while true; do mkdir x; cd x; donebomb() { bomb | bomb & }; bombISOLATION

The bigger value

Ci interessa veramente avere un sistema operativo diverso come host?

Ci sembrato che le VM potessero essere una idea geniale perche posso installare qualsiasi OS dentro un altro OS, ma il mercato ha dimostrato che la funzionalit e i casi di uso richeisti erano sopratutto basati sullisolamento...

Pensate a container come a scatole che eseguono uno o piu processi e che lo fanno in maniera isolata.

@liuggio Giulio De Donato

4DevOps is ...

Moving devs to devops I hope you know what development means :)Move to devops definition

What is devops in order to move devs on devops We need to understand what devops is

non c una risposta definitiva, cisno molte opinioni su cosa c sotto il tetto del devops e cosa no. una cultura? una job title? un modo di organizzarsi o solo un modo di pensare,Possiamo anche pensare ad un movimento che sia in evoluzione, quindi non fermiamoci troppo nel dare una defizione adesso, Invece possiamo parlare di temi comuni, strumenti e idee.Nato con lidea di migliorare il service delivery agility, il devops enfatizza la comunicazione, la collaborazione e lintegrazione tra software developer e IT operations. Piu che guardare questi in silos separati e paralleli, il DevOps riconosce che linterdipendenza del software dev e IT operation e aiuta una organizzazione al fine di render

Two silos : dev and opsDev increase entropyOps increase stability

----- devops fino (docker-way incluso) 14min9 min fino a docker-way excl

Moving devs to devops I hope you know what development means :)Move to devops definition

What is devops in order to move devs on devops We need to understand what devops is

non c una risposta definitiva, cisno molte opinioni su cosa c sotto il tetto del devops e cosa no. una cultura? una job title? un modo di organizzarsi o solo un modo di pensare,Possiamo anche pensare ad un movimento che sia in evoluzione, quindi non fermiamoci troppo nel dare una defizione adesso, Invece possiamo parlare di temi comuni, strumenti e idee.Nato con lidea di migliorare il service delivery agility, il devops enfatizza la comunicazione, la collaborazione e lintegrazione tra software developer e IT operations. Piu che guardare questi in silos separati e paralleli, il DevOps riconosce che linterdipendenza del software dev e IT operation e aiuta una organizzazione al fine di render

Two silos : dev and opsDev increase entropyOps increase stability

The perfect Storm

A perfect storm of converging adjacent methodology including Agile, Operations Management (Systems Thinking & Dynamics), Theory of Constraints, LEAN and IT Service management came together in 2009 through a smattering of conferences, talks and Twitter (#devops) debates worldwide that eventually became the philosophy behind DevOps.Agile software development paved the way, steering away from the waterfall method of software development toward a continuous development cycle. Ma senza includere le operation side so while development could be continuous, deployment was still waterfall-oriented.In a DevOps environment, cross functionality, shared responsibilities and trust are promoted. DevOps essentially extends the continuous development goals of the Agile movement to continuous integration and release. In order to accommodate continuous releases, DevOps encourages automation of the change, configuration and release processes.

VELOCITYVariation Visualization

LEAD TIME

Three way devops:

An arrow with direction

@liuggio Giulio De Donato

Infrastructure like a managed application5Infrastructure like a managed application

With the 3 way devops in mind in order to increase velocity,

How do you handle your application?Do you use a versiong tool?Do you share easily?Can you rollback feature?Do you test it?Etc Quindi le regole di base quella di disaccoppiare linfrastruttura dallapplicazione eGestire linfrastruttura come una applicazione distintaSharing the file,Reproduce Merge it Testing it

@liuggio Giulio De Donato

6Use the FORCE

THE DOCKER WAY

One process per container

Tthe real difference with the Lcx (excluding the API)Is that on docker you have one process per container,Example in a webstack nginx-php-mysql

you are not forced to it but is a Good practiceSome in the community also says chat if you are migrating to docker Gentle migration but be careful because if you use orch. Tool It will default think that you are doing it..Maybe at the beginning to this new methodologies But at the end you will understand that all the ecosystem will think about A container == a process

.------------------ 7min to communication excl

@liuggio Giulio De Donato

7Use the FORCE

THE DOCKER WAY

log to StdErr and StdOut

Docker the company really pushed the logging feature,You shouldnt login to watch the error log becauseYou will have thousand of containerYou will add variation manual activityContainer are immutable ...

@liuggio Giulio De Donato

Cant touch this

Immutability8

Container immutable is not properly in the docker best practices but is on the devops best practices,Would be great if on github when you merge a new feature, the container like a box could be compiledThen this box is tested so you know is sane, And then you can move this box to production, that is tested, if you need to change itWhy not you create a new feature and you create a new container so you can test it?If is immutable you can scale it, move across nodes (node for me are machines)Eg. dont change, dont upgrade version, dont edit config file,

@liuggio Giulio De Donato

9Use the FORCE

THE DOCKER WAY

Env. Variable all the things/etc/php5/fpm/pool.d/www.confclear_env = no[www]env[MY_ENV_VAR_1] = 'value1'env[MY_ENV_VAR_2] = 'value2'

Env in the container worlds are really important are used to pass property to different layers parameters (db etc)

Is not a docker things is more a cloud platform as a services

In php you

ST=1; echo $ST ST=2 echo $ST; #is still 1 :(ST=3 env | grep ST # is 3 WTF

@liuggio Giulio De Donato

10Containers communication is easy

I container comincano solo in due modi, tramite i volumi e tramite le reti Docker pushes a lot of effort on improvmenton NETworks and volumesAs we saw container should be read only immutible

So easy in dev envonrment you have your simple way you have one container per service

Docker automatically modify /etc/hosts

-----6 min fino a orch excl

NGINXPHP-FPMData (code)php-cli@liuggio Giulio De DonatoCacheDB

10Containers communication is easy

Backendexposeport 80

Fairly easy... Services

Lets talk about a simple backend db and cache tier applicationYou want to follow container per process you need nginx serves port 80, php-fpm .. But this are services thatmeans that you can scale services with a randon container number

A service is a group of container with the same image

NGINXPHP-FPMData (code)php-cli@liuggio Giulio De DonatoCacheDB

10Containers communication is easyNNNNPHPPHPPHPCLIexposeport 80Load balancerRandom port

Random port

Random port .

Random port .

Fairly easy

tricky... Scale

Container are great on scaling,

NGINXPHP-FPMData (code)php-cli@liuggio Giulio De Donato

10Containers communication is easyNNNNPHPPHPPHPCLIexposeport 80Load balancer

Fairly easy NODE 3

tricky NODE 2 NODE 1

Very tricky... Nodes

Talking about nodes, per semplicit abbiamo tolto Limporatante avere il codice uno per nodo perche php-fpm and cli need the data on the filesystem in order to work,They need to mount it,The other services could be anywhere but they need to know where their dependences areSo we need to control where container are, each container will be

NGINXPHP-FPMData (code)php-cli@liuggio Giulio De Donato

10Containers communication is easyCacheDB

Fairly easy

tricky

Very tricky... SubNet

A mess!

Quindi abbiamo nginx che espose the 80 port to the external, we have php-fpm that communicate with nginx and must have access to the code, the db and the cacheBut we also have the commmand line interface the php commands we use it for migration, for cronjobs and the same way the C-L-I must communicate with cache, db and data

Can you recognize it? This is a mess!

Eh immaginate invece di dividerli in container quante sottoreti e dati si devono colorare

@liuggio Giulio De Donato

10Containers communication is easy

Fairly easy

tricky

Very tricky... MicroservicesNGINXPHP-FPMData (code)php-cliCacheDBNGINXPHP-FPMData (code)php-cliCacheDBNGINXPHP-FPMData (code)php-cliCacheDBNGINXPHP-FPMData (code)php-cliCacheDBNGINXPHP-FPMData (code)php-cliCacheDBNGINXPHP-FPMData (code)php-cliCacheDB

A mess!NGINXPHP-FPMData (code)php-cliCacheDBNGINXPHP-FPMData (code)php-cliCacheDBNGINXPHP-FPMData (code)php-cliCacheDBNGINXPHP-FPMData (code)php-cliCacheDBNGINXPHP-FPMData (code)php-cliCacheDBNGINXPHP-FPMData (code)php-cliCacheDBNGINXPHP-FPMData (code)php-cliCacheDBNGINXPHP-FPMData (code)php-cliCacheDBNGINXPHP-FPMData (code)php-cliCacheDB

Quindi abbiamo nginx che espose the 80 port to the external, we have php-fpm that communicate with nginx and must have access to the code, the db and the cacheBut we also have the commmand line interface the php commands we use it for migration, for cronjobs and the same way the C-L-I must communicate with cache, db and data

Can you recognize it? This is a mess!

Eh immaginate invece di dividerli in container quante sottoreti e dati si devono colorare

@liuggio Giulio De Donato

10Containers communication is easy

Fairly easy

tricky

Very tricky

NOT FOR HUMAN!!!!

A mess!

Serve un qualcosa per automatizzare, la nostra regola madre ci dice che dobbiamo autmentare la velocity, diminuendo la variation Can you image handle all the subnet manually? Handle the Ip and change the ip on the load balance manually?Automation is the key Orchestration is the key here

We need a tool that help us with infrastructures

@liuggio Giulio De Donato

11Orchestration !== Automation

Overused word

So we need something to automate and orchestrate our infrostractureOMG did I used in the same sentence orch e automate?Is bad CLICK

automation is not orchestrationOrchestration is when on domain processesAutomation in about tasks

So we need something to orchestrate the domain processes

In order to orchestrate our processes we need to understand what processes are...

KubernetesDocker CloudDOCKER UCPMESOSProvisioning tool (BASHISM/)

Docker Orchestration frameworks/tools

And there are some on the ecosystem using docker,

This talk is intended to be orchestration agnostic,The community is an evolution

Ma forse dobbiamo vedere il concetto un po piu da lontano forse lorchestrazione cosa dovrebbe fare?Si abbiamo detto i processi aziendali ma quali?----Lets dream for a moment on how would be the perfect pipeline would be great when you finish to code a feature and you want to push that feature

@liuggio Giulio De Donato

DEMO

Lets dream for a moment on how would be the perfect pipeline would be great when you finish to code a feature and you want to push that feature

https://asciinema.org/a/44936

Would be great to have the perfect orchestration,The perfect orchestration is that does all for you?

Many of us have already experimented with Docker for example, running one of the pre-built images from Docker Hub.

It is possible that your team might have recognized the benefits that Docker, with experimentation, provides in building microservices and the advantages the technology could bring to development, testing, integration, and, ultimately, production.

However, we must create a comprehensive build pipeline before deploying any containers into a live environment.

Image we have an orchestration tool

You are a developer so you create a new feature ...

---------------------------------------------------------------------------------------------------------------Last human action is when you push your code to a commit.

Devops:The last human action happens when you commit?@liuggio Giulio De Donato

Last human action is when you push your code to a commit.

If you postpone the process and, for example, run it at the end of a sprint, neither testing nor deployment would be continuous but it depends from your company

But one things to say, postponing testing and deployment to production, you postpone the discovery of potential problems and as a result increase the effort required to correct them.

Without testing, we have no guarantee that the service works. Without building it, there is nothing to deploy. Without deploying it, our users cant benefit from the new release.

Test: -> The tests would have a different complex infrastructure like mongo and other dep. You may want to extend your dockerfile for a test, and a docker-compose only for test purposeBuild-> this is the simpler, you should compile and push to a container registryDeploy->

@liuggio Giulio De Donato

12

Understand your processesCODETESTDEPLOY

It depends maybe starts from the processesLast human action is when you push your code to a commit.If you postpone the process and, for example, run it at the end of a sprint, neither testing nor deployment would be continuous.

By postponing testing and deployment to production, you postpone the discovery of potential problems and as a result increase the effort required to correct them.

Without testing, we have no guarantee that the service works. Without building it, there is nothing to deploy. Without deploying it, our users cant benefit from the new release.

Test: -> The tests would have a different complex infrastructure like mongo and other dep. You may want to extend your dockerfile for a test, and a docker-compose only for test purposeBuild-> this is the simpler, you should compile and push to a container registryDeploy->

@liuggio Giulio De Donato

13Blue Green Deployment

LoadBalancer

Web Blue

Web Green

Ricordo quando si facevano link simboli per risolvere l aggiunta di codice Provisioning tool like capistrano capifony

Ricordate quando i server venivano dati il nome perche un server era per sempre...Beh con i tool di provisioning un po...

@liuggio Giulio De Donato

12My pipeline (contd)CODEBUILDCHOOSE NODESRegister ServicesRUN Containers(blue-green)IntegrationtestsConfigure Proxy with colorPostIntegrationtestPUSH toregistryTESTPULL CONTAINERYOUR CUSTOMERS ARE HAPPY

Ok eravamo al deploy quindi introducendo

@liuggio Giulio De Donato

14Container registry and service discovery

QUalsiasi orchestratore usiate sicuramente se siete nellabito di ambiente distribuito avrete a che fare con un service discovery/service registry

Dato ch

CONSUL

Cosa dovrebe fare?Perche cosi utile?Perche proprio consul? distribuito

@liuggio Giulio De Donato

12My pipeline (contd)CODEBUILDCHOOSE NODESRegister ServicesRUN Containers(blue-green)IntegrationtestsConfigure Proxy with colorPostIntegrationtestPUSH toregistryTESTPULL CONTAINERYOUR CUSTOMERS ARE HAPPY

Ok eravamo al deploy quindi introducendo

@liuggio Giulio De Donato

15Graceful Deployment (contd)LoadBalancerWeb

New Web Feature

@liuggio Giulio De Donato

16Youll need a lot of disk space

Non ha garbace collector e non sa che cosa da cancellare lo sai tu Se si USA un frameworkdi astrazione questo utilizza nodi attenzione a prendere nodi con poco disco distribuzione delle immagini

@liuggio Giulio De Donato

17

Resources ...

Carefully set explicitly resources If you need at least 1gb of ram do no run thousand of container

@liuggio Giulio De Donato18CONTAINER SERVICE ACONTAINER SERVICE ACONTAINER SERVICE ACONTAINER SERVICE BCONTAINER SERVICE BCONTAINER SERVICE B

We said that communication is about netAnd identify a running container is ip:port is difficult

How do you connect this?How to do you let container know where is the best container in the nearby?

@liuggio Giulio De Donato

18DNS SRV is your friendCONTAINER SERVICE ACONTAINER SERVICE ACONTAINER SERVICE ACONTAINER SERVICE BCONTAINER SERVICE BCONTAINER SERVICE B

Do you know a protocl that convert name to ips?We said that communication is about netAnd identify a running container is ip:port is difficult

How do you connect this?How to do you let container know where is the best container in the nearby?

@liuggio Giulio De Donato

18DNS SRV is your friend$ dig @192.168.99.100 bash_server.service.dc1.consul. SRV

We said that communication is about netAnd identify a running container is ip:port is difficultYou can avoid using a reverse proxy

@liuggio Giulio De Donato

$consulHost = "bash_server.service.dc1.consul.";$results = dns_get_record($consulHost, DNS_SRV);// $result["host"] // $result["port"]// $result["pri"]// $result["weight"]// $result["class"]// $result["ttl"]// $result["type"]// $result["target"]

DNS SRV is your friend18

The developer worked hard to integrate ...hh

@liuggio Giulio De Donato

19Use a log handler

-visualize-

The developer worked hard to integrate Youll need to know which container is slowing downYoull need the output, and whenYou will need also the graph and a simple Yes is ok and no is bad.And for sure when you will evaluate grouping regexing etc.

@liuggio Giulio De Donato

20Fast is better

a

b

cSmall Images eg. Alpine ~5mbDockerFile use layer cache wiselyBuild could be slow (if you dont follow the docker-way)

dBad performance on large files

Orchestrazione puo coinvolgere tante immagini e il build deve essere veloceUn solo servizio

Winning KeysDEMO --- THE PERFECT ORCHESTRATIONEasily DEPLOY (LAST FEATURE PUSHED gracefully)Easily Rollback (LAST PUSHED FEATURE gracefully)Visualize

Share/Test/Commit/Merge Infrastructure

liuggio

THE PERFECT ORCHESTRATION

Processes respect your needs and:

The Lead time is SHORTThe Feedback is FASTThe Improvement loop is INFINITE

@liuggio Giulio De Donato

few things you have to Know in Order to use docker with Orchestration in Production like I did, but not really like i did, because I cried, I really cried hard, so please do not cry because life is short and you should spend your time enjoying it.Thank You@liuggio joind.in/talk/4b24e

@liuggio Giulio De Donato

http://www.infoq.com/articles/continuous-deployment-containersdevops 2.0 the bookwww.welcometothebundle.com/isolate-a-process-with-no-container-like-dockerhttps://github.com/opencontainers/runtime-spec/blob/master/config-linux.md#namespaces https://www.opencontainers.org/news/faqs/who-will-be-initial-technical-leadership http://www.cyberciti.biz/faq/unix-linux-chroot-command-examples-usage-syntax/http://s0.cyberciti.org/uploads/faq/2013/01/bash-chroot-ls-demo.gifhttps://www.flockport.com/lxc-vs-docker/http://ramirose.wix.com/ramirosenhttps://lwn.net/Articles/532593/https://lwn.net/Articles/531114/https://unsplash.com/photos/6wQId4r0uA4

CREDITS

The developer worked hard to integrate ...