從 Ansible 到 Docker:混血模式 (Build Docker images with Ansible: A half-blood approach)
Docker, Ansible and practical experiences....Ansible and Docker • Two relevant modules for Ansible...
Transcript of Docker, Ansible and practical experiences....Ansible and Docker • Two relevant modules for Ansible...
PRODUCTS • CONSULTING • APPLICATION MANAGEMENT • IT OPERATIONS • SUPPORT • TRAINING
Docker, Ansible and practical experiences.
Kristian Lyngstøl, Redpill LinproOslo, June, 2016
PRODUCTS • CONSULTING • APPLICATION MANAGEMENT • IT OPERATIONS • SUPPORT • TRAINING
Background
• Senior Systems Consultant for Redpill Linpro
• Volunteer at The Gathering
• Worked with:
• Docker for 3 years
• Configuration management for 7 years
• GNU/Linux for 17 years
• C/Perl/PHP/Python/JavaScript/Basic/AWK/+++ for 23 years.
• A well established knack for finding weird and obscure bugs.
• Trust issues.
PRODUCTS • CONSULTING • APPLICATION MANAGEMENT • IT OPERATIONS • SUPPORT • TRAINING
PRODUCTS • CONSULTING • APPLICATION MANAGEMENT • IT OPERATIONS • SUPPORT • TRAINING
Docker in one sentence
A client/server implementation for building and maintaining file archives
and for running them in isolated environments.
PRODUCTS • CONSULTING • APPLICATION MANAGEMENT • IT OPERATIONS • SUPPORT • TRAINING
What?
• Define an image in a Dockerfile using simple commands (Start FROM this image, ADD this file, RUN this command, EXPOSE this port, etc)
• Upon changing a Dockerfile, only a small part of the image has to be rebuilt.
• Run as many copies of said image as you want. Each running instance is a container.
• Any changes made in a container is gone if you restart it. Sort of.
PRODUCTS • CONSULTING • APPLICATION MANAGEMENT • IT OPERATIONS • SUPPORT • TRAINING
Dockerfile
FROM debian:jessie
RUN aptget update
RUN aptget y install varnish
RUN rm /etc/varnish/default.vcl
ADD extras/misc/varnish.vcl /etc/varnish/default.vcl
CMD varnishd a :80 f /etc/varnish/default.vcl F
EXPOSE 80
PRODUCTS • CONSULTING • APPLICATION MANAGEMENT • IT OPERATIONS • SUPPORT • TRAINING
Docker pros
• Great for development and testing
• Enforces a well defined environment (sort of)
• “Just works”
• Lighter than a virtual machine
• Theoretically better hand-off to operations
PRODUCTS • CONSULTING • APPLICATION MANAGEMENT • IT OPERATIONS • SUPPORT • TRAINING
Docker cons
• Overly pragmatic. Immature
• Still IPv4-centric in 2016.
• Hides complexity and dependencies. Black boxes everything.
• Infantile approach to logging.
• Obvious development work flow. Non-obvious production work flow.
• No obvious non-disruptive way to combine with existing applications.
• Requires considerable commitment to utilize fully
PRODUCTS • CONSULTING • APPLICATION MANAGEMENT • IT OPERATIONS • SUPPORT • TRAINING
The big one: Systemd++
• Horrible “religious war” between Docker developers and systemd developers.
• Just running 5-20 containers shouldn't require any real orchestration at all, but does.
• Makes integration with existing systems a pain.
• Forces an “all or nothing” situation
PRODUCTS • CONSULTING • APPLICATION MANAGEMENT • IT OPERATIONS • SUPPORT • TRAINING
PRODUCTS • CONSULTING • APPLICATION MANAGEMENT • IT OPERATIONS • SUPPORT • TRAINING
Ansible in one sentence
Minimalistic way to perform well-defined tasks across multiple
machines.
PRODUCTS • CONSULTING • APPLICATION MANAGEMENT • IT OPERATIONS • SUPPORT • TRAINING
What?
• No centralized controller. No daemon. Just command line.
• Write a YAML fine defining tasks that are potentially linked. (“Install this package repository”, “Enable this service”, “Build this docker container”)
• Write an inventory. Can be a single hostname in a file.
• Run ansible-playbook and it will connect to relevant hosts as needed.
PRODUCTS • CONSULTING • APPLICATION MANAGEMENT • IT OPERATIONS • SUPPORT • TRAINING
Ansible playbook:
hosts: all
become: true
tasks:
apt: name=nano state=absent
git: repo=https://github.com/techserver/gondul.git
dest=/opt/gondul
PRODUCTS • CONSULTING • APPLICATION MANAGEMENT • IT OPERATIONS • SUPPORT • TRAINING
Ansible pros
• Extremely easy to get started.
• Can be combined with existing configuration management or orchestration.
• Doesn't require dramatic changes.
PRODUCTS • CONSULTING • APPLICATION MANAGEMENT • IT OPERATIONS • SUPPORT • TRAINING
Ansible cons
• Simple. For better or worse: Just tool.
• Replacing existing configuration management can be tricky
• Performance issues if you have many nodes (Python and parallelism isn't a match made in heaven)
• Several modules are somewhat immature
PRODUCTS • CONSULTING • APPLICATION MANAGEMENT • IT OPERATIONS • SUPPORT • TRAINING
The Docker challenge
• How do you actually run these things in production?
• Lack of systemd-integration makes things less smooth than it could have been. Specially for smaller projects.
• Many emerging tools, all requiring big commitments
• Mesos, kubernetes, openshift?
• Basically, things haven't really settled yet.
PRODUCTS • CONSULTING • APPLICATION MANAGEMENT • IT OPERATIONS • SUPPORT • TRAINING
Ansible and Docker
• Two relevant modules for Ansible 2.0.x
• “docker_image” to build an image
• “docker” to run a container (off of an image)
• Ansible 2.1.0 (released recently) expands and improves this. Notably introducing support for “docker compose”.
• Works well for smaller installations
• Everything can be kept in a single git repository
• Can be used for larger installations, but requires you to do a lot of thinking that others have done for you in competing solutions.
PRODUCTS • CONSULTING • APPLICATION MANAGEMENT • IT OPERATIONS • SUPPORT • TRAINING
Short Intermission: The Gathering
• Annual non-profit digital festival attracting 6000-8000 people for 5 days each easter for gaming, concerts, socializing, esports, cosplay, shows and more.
• 400 volunteers, about 35 of which are part of the technical crew.
• 10,500 MAC addresses, 300-400 switches, access points and routers.
• Great test-bed for technology.
• http://www.gathering.org
PRODUCTS • CONSULTING • APPLICATION MANAGEMENT • IT OPERATIONS • SUPPORT • TRAINING
Pictures courtesy of Joachim Tingvold
PRODUCTS • CONSULTING • APPLICATION MANAGEMENT • IT OPERATIONS • SUPPORT • TRAINING
A few days earlier...
PRODUCTS • CONSULTING • APPLICATION MANAGEMENT • IT OPERATIONS • SUPPORT • TRAINING
Various fiber cables Meru Wireless Access Points
Servers we need in production within 24-96 hours
Pictures courtesy of Joachim Tingvold
PRODUCTS • CONSULTING • APPLICATION MANAGEMENT • IT OPERATIONS • SUPPORT • TRAINING
Pictures courtesy of Joachim Tingvold
About 200 Juniper EX2200 48-port switches
PRODUCTS • CONSULTING • APPLICATION MANAGEMENT • IT OPERATIONS • SUPPORT • TRAINING
A second Juniper QFX10002-72q being installed($200,000-$250,000 each, 72 40Gbit ports or 24 100GBit ports)
Pictures courtesy of Joachim Tingvold
PRODUCTS • CONSULTING • APPLICATION MANAGEMENT • IT OPERATIONS • SUPPORT • TRAINING
~400 Volunteers
Pictures courtesy of Joachim Tingvold
PRODUCTS • CONSULTING • APPLICATION MANAGEMENT • IT OPERATIONS • SUPPORT • TRAINING
My job: Providing tools needed to ensure a good technical delivery.
PRODUCTS • CONSULTING • APPLICATION MANAGEMENT • IT OPERATIONS • SUPPORT • TRAINING
Monitoring: Gondul
• Custom built to match the work flow and demand.
• Simple and robust design.
• Postgres database, 3-4 stand-alone data collectors, REST API, Varnish cache, JavaScript/HTML frontend.
• Used for other events too – open source.
• Served around 600 HTTP requests per second during TG16. About 300-400 rows inserted per second.
• Delay from data collection to visibility typically sub-second.
(Hi, Dream Hack).
PRODUCTS • CONSULTING • APPLICATION MANAGEMENT • IT OPERATIONS • SUPPORT • TRAINING
Differences between Gondul and other projects
• Only needs to run it for 9 days.
• Simulating traffic from 7000 concurrent devices ranging from cell phones to laptops and x-boxes mixed with DDoS attacks is impossible.
• Can't force people to do boring things. If it isn't motivating at some level, it wont get done.
• Hard to introduce new developers to the project until they've seen how it's used.
PRODUCTS • CONSULTING • APPLICATION MANAGEMENT • IT OPERATIONS • SUPPORT • TRAINING
Development and deployment
• Plan for changes: Development ahead of the event has to anticipate rapid changes to code and physical infrastructure.
• Self-documentation important.
• Easy (re-)deployment is key.
• Ridiculously overpowered hardware provided by gracious sponsors has to be utilized.
• Docker was an attractive alternative.
PRODUCTS • CONSULTING • APPLICATION MANAGEMENT • IT OPERATIONS • SUPPORT • TRAINING
Attempt 1: Docker + systemd + homebrew
• Made transition between development in containers to deployment on hardware easier
• Largely self documenting
• Systemd integration was absolutely horrible and varied wildly between developers' individual systems.
• Nearly made me deaf when a docker container decided to reset my “host” volume to 100%.
• Development during the event was efficient, but awkward.
PRODUCTS • CONSULTING • APPLICATION MANAGEMENT • IT OPERATIONS • SUPPORT • TRAINING
Attempt 2: Docker + Ansible
• Cleaner Dockerfiles (no systemd integration)
• Closer coupling to containers.
• No homebrew == leverage existing Ansible modules to get bonus features
• Finally getting some much needed test cases
• Development can still take place during the event and ahead of time
• Production database non-containerized. Development database containerized.
PRODUCTS • CONSULTING • APPLICATION MANAGEMENT • IT OPERATIONS • SUPPORT • TRAINING
Demo
Example: Building
hosts: all
become: false
vars:
images:
name: "nmsdbtest"
links: []
name: "nmsfronttest"
links: [ "nmsdbtest:db" ]
name: "nmsvarnishtest"
links: [ "nmsfronttest:nmsfront" ]
name: "nmscollectortest"
links: [ "nmsdbtest:db" ]
tasks:
name: make all
docker_image:
state: build
name: "{{ item.name }}"
dockerfile: /opt/nms/build/test/{{ item.name }}.Dockerfile
path: "/opt/nms"
with_items: "{{ images }}"
Example: Basic tests
name: workaround to get nmsvarnishfrontip
shell: "docker inspect nmsvarnishtest | grep IPAddress | sed 's/[^09.]//g'"
register: ip
name: Display IP
debug:
msg: "Varnish test is available at http://{{ ip.stdout }}/"
name: test index
uri: url="http://{{ ip.stdout }}/"
name: test public api without data
uri:.
url: "http://{{ ip.stdout }}{{ item }}"
with_items: "{{ simple_urls }}"
PRODUCTS • CONSULTING • APPLICATION MANAGEMENT • IT OPERATIONS • SUPPORT • TRAINING
Immaturity
• Neither Docker nor Ansible's docker modules can give me the IP reliably.
• “IPAdress” in docker still means legacy-IP, because why be part of the solution when you can be part of the problem.
• The docker “fact” of ansible has issues, even in recent 2.1.0.
• Specifically: Only shows facts for one container.
• Not accessible in a pretty way (e.g.: a list, not a dictionary)
• I assume these issues will be resolved with time
PRODUCTS • CONSULTING • APPLICATION MANAGEMENT • IT OPERATIONS • SUPPORT • TRAINING
Deploying?
Just run it on a different host. Simple run-time configuration for picking
database server.
PRODUCTS • CONSULTING • APPLICATION MANAGEMENT • IT OPERATIONS • SUPPORT • TRAINING
Potential and challenges
• Persistent data in a container is still painful, but possible.
• Being able to use the same solution to deploy non-docker infrastructure is a nice benefit.
• Bootstrapping the host.
• Experimenting with using Ansible to control and at least partially configure network equipment.
• If we used persistent infrastructure: Use a registry to do handover from development to production.
PRODUCTS • CONSULTING • APPLICATION MANAGEMENT • IT OPERATIONS • SUPPORT • TRAINING
Questions?