Building a PaaS with Docker, Consul and Python
-
Upload
konstantin-nazarov -
Category
Technology
-
view
857 -
download
4
Transcript of Building a PaaS with Docker, Consul and Python
![Page 1: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/1.jpg)
Building a PaaS with Docker, Consul and Python
by Konstantin Nazarov
![Page 2: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/2.jpg)
![Page 3: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/3.jpg)
I work in Tarantool And I've built a PaaS this way
![Page 4: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/4.jpg)
I'll share how to build one• It is simple
• The experience is highly portable
• You can start small and grow iteratively
• Fits your requirements
• The tech stack is widely known
![Page 5: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/5.jpg)
1. docker host
2. docker host
3. orchestrator
4. consul
5. web UI
![Page 6: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/6.jpg)
Why?
![Page 7: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/7.jpg)
Why build your own PaaS?
• If you develop your own product and sell it
• Small initial investment to solve lifecycle problems
• A way to fit your requirements exactly
• Keep your operations lean
• To enable fast experiments
![Page 8: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/8.jpg)
When not to build it
• When you run off-she-shelf software
• If you have large monolithic services
![Page 9: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/9.jpg)
What a PaaS should do
• Run the code you give to it
• Abstract away the OS
![Page 10: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/10.jpg)
Let's build it
![Page 11: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/11.jpg)
• We will build progressively
• On each step there will be a working system
Let's build it
![Page 12: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/12.jpg)
A few building blocks
![Page 13: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/13.jpg)
1. Python• Because it is simple
• There are bindings for almost all existing stuff
![Page 14: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/14.jpg)
2. Docker
• Has remote HTTP API to run stuff
• Has convenient packaging format
![Page 15: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/15.jpg)
2. Docker: the good
• Simple
• Well documented
![Page 16: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/16.jpg)
2. Docker: the bad
• Bugs
• Weak networking
![Page 17: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/17.jpg)
2. Docker: Alternatives
• fabric (yes, as an RPC)
• gearman
• nomad
![Page 18: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/18.jpg)
3. Consul
• Fault-tolerant key-value storage
• Service registry with active checks
• Easily deployed
![Page 19: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/19.jpg)
3. Consul: alternatives
• etcd
• zookeeper
![Page 20: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/20.jpg)
MK1: Smart command-line
client to Docker/Consul
![Page 21: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/21.jpg)
Why?
• quick to implement
• very high value compared to effort
![Page 22: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/22.jpg)
What it should do
• run
• inspect
• upgrade
• rm
• ps
![Page 23: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/23.jpg)
Example usage
$ mypaas run git://gitserver/project.git v1.2
fddf3f
$ mypaas upgrade fddf3f v1.3
$ mypaas rm fddf3f
![Page 24: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/24.jpg)
How
• Docker API is exposed on physical servers
• Physical servers are registered in consul
• CLI connects to a known consul host
• Docker API is used to build app container "in place"
![Page 25: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/25.jpg)
How to choose physical nodes
• By maximum memory requirements
• By conventional CPU units
• By number of services running
![Page 26: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/26.jpg)
Upgrading versions
• Stop the running container
• Start new container inheriting volumes from the old
• On success, remove old container
• On failure, restart old container
![Page 27: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/27.jpg)
Result
• A working PaaS
![Page 28: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/28.jpg)
Improving MK1: health checks
• Consul API can be used to register health checks
• Consul can run commands in docker containers
• The CLI can poll consul for service statuses
![Page 29: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/29.jpg)
Wiring things together via network
![Page 30: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/30.jpg)
Docker networking hell
• There are overlay networks (UDP encapsulation)
• And macvlan (adding new MAC addresses to eth)
• And openvswitch with DPDK
• And god knows what else (BGP routing anyone?)
![Page 31: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/31.jpg)
Let's use plain bridges
br0
![Page 32: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/32.jpg)
Let's use plain bridges
• Docker IPAM doesn't know about other nodes
• IP conflicts are possible
• So we have to use our own IPAM
• Write allocated IPs to consul KV
• Set IPs explicitly
![Page 33: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/33.jpg)
IPAM
![Page 34: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/34.jpg)
MK2: running as a service
![Page 35: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/35.jpg)
Why?• Limiting access to production servers
• Active monitoring of business logic
• Concurrent access
![Page 36: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/36.jpg)
How?
• flask
• flask-restful
• gevent
![Page 37: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/37.jpg)
Why these tools?
• Everything fits in one app
• Orchestration code is easier to write in async mode
• Quick to implement
• Your CLI becomes an HTTP API client
![Page 38: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/38.jpg)
Separate state• Orchestrator itself should be stateless
• Orchestrator should show the system overview
• You probably need basic auth at this step
![Page 39: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/39.jpg)
MK3: admin UI
![Page 40: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/40.jpg)
Why?
• Easier to manage
• Easier to debug
![Page 41: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/41.jpg)
Example
![Page 42: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/42.jpg)
How?
• flask templates
• bootstrap
![Page 43: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/43.jpg)
What to have here
• Overview of physical servers and their state
• Overview of your services and their state
• CRUD
![Page 44: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/44.jpg)
MK4: delayed tasks and active checks
![Page 45: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/45.jpg)
Why?
• Distributed cron
• Data extraction
• Backups
![Page 46: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/46.jpg)
Tools
• gevent
• Docker exec API
![Page 47: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/47.jpg)
How?
• On server start, spawn a worker fiber
• In the fiber, poll consul and run your code
• Or start worker fibers on demand
• Send notification emails upon completion
![Page 48: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/48.jpg)
MK5: metrics and time series
![Page 49: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/49.jpg)
Why?
• Historical data matters for problem solving
• See how well your new code is behaving over time
![Page 50: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/50.jpg)
How?
• install prometheus and hook it up to consul
• use prometheus API to query aggregates
![Page 51: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/51.jpg)
Exporting metrics to prometheus
• Either add support to your service
• Prometheus protocol is very simple!
• Or collect via the orchestrator
![Page 52: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/52.jpg)
![Page 53: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/53.jpg)
Recap
![Page 54: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/54.jpg)
Recap• docker-python - running stuff
• python-consul - storing information about running stuff
• flask - serving admin UI
• flask-restful - providing HTTP API
• bootstrap - making your web page less ugly
• gevent - running delayed tasks and async code
• prometheus - storing time series
![Page 55: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/55.jpg)
![Page 56: Building a PaaS with Docker, Consul and Python](https://reader033.fdocuments.net/reader033/viewer/2022052318/587bf95c1a28ab7c668b5211/html5/thumbnails/56.jpg)
Thanks!Konstantin Nazarov
Building orchestrators is not that hard.
@racktear
http://bit.ly/paas-bom