Building Docker Containers @ Scale
-
Upload
lxfontes -
Category
Engineering
-
view
544 -
download
4
Transcript of Building Docker Containers @ Scale
![Page 1: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/1.jpg)
Building containers
Go-TorontoNov/2016
@lxfontes
![Page 2: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/2.jpg)
#bfcm
![Page 3: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/3.jpg)
Continuous Delivery Pipeline 1-1
BuildGit Push Test Deploy
![Page 4: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/4.jpg)
Your company keeps growing
![Page 5: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/5.jpg)
Growing
![Page 6: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/6.jpg)
Growing MOAR...
![Page 7: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/7.jpg)
![Page 8: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/8.jpg)
Clone repository
docker build
- Install RubyGems
- Install NPM Packages
- Compile static assets
- Seal Docker image
docker push
- Upload to Docker Registry
Building a Rails Container
![Page 9: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/9.jpg)
InertiaYou get used to it
Your build time only goes up
Number of workers only goes up
“Our build is complex”
“We already use Docker, the Cloud. This is as fast as it gets!”
“It is automated, we don’t care it takes 30 minutes”
“The XYZ team should fix it”
![Page 10: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/10.jpg)
it was a reality check20 minutes per build
Flakiness
Resource Starvation
Really expensive
* where do you put secrets??????
![Page 11: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/11.jpg)
Why is it slow?
push webhook
worker bootstrap
docker build cache not reused between builds
cold asset precompilation
2nd build
![Page 12: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/12.jpg)
peed
![Page 13: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/13.jpg)
Where are we: Daily stats
+200Devs
+700Builds
+42kTests
per build
~3Images
per build
![Page 14: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/14.jpg)
Commitment
Container
BuildGit Push
Automated
TestsDeploy
5s 90s 200s 300s
From Git Push to Deployed in ~10 minutes
Every developer can deploy to productionGoals:
Testing Rails at
Scale by Emil
How We Deploy
Shopify by KatThis right here
![Page 15: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/15.jpg)
push 30 sec POLL
Pull / Build
Coordinator+ API
zookeeper
workers
registry
branch affinity
![Page 16: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/16.jpg)
ohai bootstrap
![Page 17: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/17.jpg)
Clone repository
docker build
- Install RubyGems
- Install NPM Packages
- Compile static assets
- Seal Docker image
docker push
- Upload to Docker Registry
Building a Rails Container
![Page 18: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/18.jpg)
Dockerfile
![Page 19: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/19.jpg)
mo layers, mo problems
* apparently docker 1.13can squash these
![Page 20: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/20.jpg)
Look ma, no Dockerfilesprepare
Install OS packages
precompile
Load (ejson) secrets
Populate artifact cache
compile
Combine app + artifact cache
Seal image
Daily seed 20min
Builds 2minInheritCache
InheritCache
![Page 21: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/21.jpg)
Docker LayersBase
Ubuntu-ish
Prepare
OS packages
Intermediate (Daily seed)
App + Libraries
Final
Delta between Intermediate … Current
![Page 22: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/22.jpg)
Locutus trade offsFAST
Local caches
Secure
Stable
But needs love
We need to maintain:
Infrastructure
Orchestration
Web UI
Scripts Copy & Pasta
Dockerfiles / Buildpacks :(
Troubleshooting :(
![Page 23: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/23.jpg)
Pipa* Kite in Portuguese(and sounds cool)
![Page 24: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/24.jpg)
Provider agnostic
Disposable
Secure
Sandboxed Docker Daemon (dind)
Pre-made recipes
Buildpack
Dockerfiles
Locutus Assembly
Pipa: RequirementsBuild whatever you want
Cache whatever you want
Repeatable
Run locally if needed
As fast as Locutus
![Page 25: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/25.jpg)
Is there an app fo dat?Orchestration / UI
Docker first
On-premise
Worker fully sandboxed
Concurrency Primitives
“Only 3 builds for app X at same time”
Parallelism
“Send job to 30 workers”
![Page 26: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/26.jpg)
BuildkiteOn-premise
Orchestration *only*
Github Hooks
UI / Reporting
API
Agent is a single binary (Go)
Also distributed as docker container
Concurrency Primitives
Parallelism
Per Build artifacts
Per Build shared key-value store
Switch workers!!
No batteries included … on purpose
![Page 27: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/27.jpg)
Buildkite Pipeline Sample - buildkite/sample-pipelines
Waits for previous stepto complete
Requires InteractionUI/API/slack
Branch filtering
![Page 28: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/28.jpg)
Can it scale?
![Page 29: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/29.jpg)
BuildkiteGitHub BK PipaDocker
RegistryBK Tests
Webhook Build Container
Switch Worker
Start Tests
Fetch Image
Tests DoneShip It!
Tests
Upload Image
100 cores 8820 coresPipa: Goal
![Page 30: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/30.jpg)
Pipa: Architecture goalsEverything exposed via environment variables
PIPA_SSH_KEY / PIPA_APP_COMMIT_ID
Chainable (simplify IF/ELSE ops)pipa build --unless v1 -- docker build -t myimage:v1 .
Configurable via environment, config file, argument switchesPIPA_APP_NAME=myapp pipa build ...
pipa build --app-name myapp …
Isolated Docker Daemon (docker-in-docker)Reset after each buildKubernetes: external SSD / Local: give it a directory
![Page 31: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/31.jpg)
Environment / Process Tree
pipa
wrapper
AWS / GCS / SSH Keys
Kubernetes secretsenv vars with paths
APP Name
Buildkite / Jenkins / etcGIT (local)
APP SHA
Buildkite / Jenkins / etcGIT (local)
Your code
Signals
![Page 32: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/32.jpg)
Prototype: I’m going to write this all in bash
![Page 33: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/33.jpg)
Single Binary
Shell files as Assets
using go-bindata
docker / aws-cli / rsync
![Page 34: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/34.jpg)
![Page 35: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/35.jpg)
LOL
![Page 36: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/36.jpg)
BUT, normalized environment is solid!
![Page 37: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/37.jpg)
Provider agnostic
Disposable
Secure
Sandboxed Docker Daemon (dind)
Pre-made recipes
Buildpack
Dockerfiles
Locutus Assembly
Pipa: RequirementsBuild whatever you want
Cache whatever you want
Repeatable
Run locally if needed
As fast as Locutus
![Page 38: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/38.jpg)
Toolbox: spf13/cobraCLI handlingdocker, kubernetes, rkt, etcd
ABUSE: func init()
![Page 39: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/39.jpg)
Toolbox: kelseyhightower/envconfig *spf13/cobra deficiency
Fills struct with env vars
![Page 40: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/40.jpg)
Why both?
spf13/cobra
kelseyhightower/envconfig
![Page 41: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/41.jpg)
Result
Google Cloud Storage
Amazon S3
JSON Asymmetric Encryption
Shopify/ejson
Registry Interaction
push / pull / exists
mmap diff 2 dirs
burke/treediff
Environment normalizer (entrypoint)
![Page 42: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/42.jpg)
Using itEverything is namespaced!
pipa cache pull /tmp/artifacts
pipa build --if base -- docker build -t $PIPA_IMAGE_FULL_NAME .
pipa image exists --local --tag basepipa image exists --remote --tag base
Downloads s3://artifacts/app_group/app_env/app_name/cache.tar to /tmp/artifacts
Only runs command if registry/app_group/app_env/app_name:base exists
Checks if registry/app_group/app_env/app_name:base exists (locally/remotely)
![Page 43: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/43.jpg)
Pipeline InferenceHow should pipa build your app?
Customize it in your repo; or
Select a template; or
Let pipa figure it out!
![Page 44: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/44.jpg)
Built-in pipelines
![Page 45: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/45.jpg)
Pipeline Output
![Page 46: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/46.jpg)
PipelineParallelization
3 parallel builds
![Page 47: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/47.jpg)
Pipeline SelectionCustom script in repo
Custom pipeline in repo
Default pipeline in repo
User requested specific pipeline
Check for common file locationsrepo/Dockerfile? -> dockerrepo/borg? -> borg
Default to Heroku Buildpack
PIPA_PIPELINE_CMD
PIPA_PIPELINE_FILE
PIPA_PIPELINE_TEMPLATE
repo/.buildkite/pipeline.yml
![Page 48: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/48.jpg)
High Level
push webhook
Agent pollSSL
dockerimages
cacheartifacts
QueueAPI
User Interface
![Page 49: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/49.jpg)
ssh-keygcloud-keybuildkite-tokendocker-auth
Sec: Kubernetes Layout
EJSONpriv/pub keypairs
/builder-secrets
/app-secrets
/var/lib/docker (dind)
/builds (git clone)
/cache (artifacts)
6x nodes
24 pods
/mnt/disks/ssd0/builder1/docker
![Page 50: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/50.jpg)
Provider agnostic
Disposable
Secure
Sandboxed Docker Daemon (dind)
Pre-made recipes
Buildpack
Dockerfiles
Locutus Assembly
Pipa: RequirementsBuild whatever you want
Cache whatever you want
Repeatable
Run locally if needed
As fast as Locutus?
![Page 51: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/51.jpg)
Gems
![Page 52: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/52.jpg)
![Page 53: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/53.jpg)
Tar + Gzip
![Page 54: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/54.jpg)
Go + Docker = fsouza/go-dockerclient
Respect environment variables
Reads $HOME/.docker/config.json or $HOME/.dockercfg
![Page 55: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/55.jpg)
Rocker - grammarly/rockerDockerfile on steroids
![Page 56: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/56.jpg)
Herokuish - gliderlabs/herokuishIf you like Heroku, you will want this
Packaged as docker container or single binary
![Page 57: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/57.jpg)
alpinelinux.org
based on musl libc and busybox
Lots of *up-to-date* packages (yes debian, looking at you)
![Page 58: Building Docker Containers @ Scale](https://reader034.fdocuments.net/reader034/viewer/2022052309/5876b50f1a28abad1a8b5807/html5/thumbnails/58.jpg)
Demo Time