Parallel Computing: Perspectives for more efficient hydrological modeling
Efficient Parallel Testing with Docker
-
Upload
laura-frank -
Category
Technology
-
view
2.190 -
download
2
Transcript of Efficient Parallel Testing with Docker
![Page 1: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/1.jpg)
Efficient Parallel Testing with Docker
Laura FrankSenior Engineer, Codeship
![Page 2: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/2.jpg)
![Page 3: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/3.jpg)
![Page 4: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/4.jpg)
1. Parallel Testing: Goals and Benefits
2. DIY with LXC
3. Using Docker and the Docker Ecosystem
Agenda
![Page 5: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/5.jpg)
Parallel Testing
![Page 6: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/6.jpg)
Create a customizable, flexible test environment that enables us
to run tests in parallel
GOAL
![Page 7: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/7.jpg)
Spend less time waiting around for your automated testing steps and deployments to finish.
• Ship newest code to production faster • Be alerted quickly when tests fail
Why?
If you’re still not sure why testing is important, please talk to me at the Codeship booth.
![Page 8: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/8.jpg)
Developers should have full autonomy over testing environments, and the way tests are executed.
• Move testing commands to separate pipelines • Designate commands to be run serially or in parallel • Declare specific dependencies for each service
Why?
![Page 9: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/9.jpg)
Let’s eliminate friction in the development cycle
![Page 10: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/10.jpg)
• For local testing, e.g. unit and integration tests run by a development team
• On internal CI/CD systems
• As part of a hosted CI/CD solution (like Codeship)
Where?
![Page 11: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/11.jpg)
• Split up testing tasks
• Give developer full control of dependencies per task
• Use containers to run multiple tests at once
How?
![Page 12: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/12.jpg)
![Page 13: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/13.jpg)
Run tasks across multiple processors in parallel
computing environments
TASK PARALLELISM
![Page 14: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/14.jpg)
Distributed Task Parallelism
A distributed system of containerized computing environments takes the place of a single multiprocessor machine
A container is a process, not a small VM
![Page 15: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/15.jpg)
Why not VMs?• Isolation of running builds on infrastructure
• Challenges with dependency management
• No clean interface for imposing resource limits
• Infrastructure is underutilized which makes it expensive
![Page 16: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/16.jpg)
Containers, duh!• Impose resource limits and increase
virtualization density
• Run customer code in isolation
• Provide consistent build environment across many build runs
• Run testing tasks in parallel 👍
![Page 17: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/17.jpg)
DIY with LXC
![Page 18: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/18.jpg)
Codeship has been powered by containers since the very beginning
![Page 19: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/19.jpg)
Codeship was founded
Flowing salty water on Mars
International Year of Forests
Preparations ahead of 12.04 Precise release shipped to support LXC improvements
Green Bay Packers won Super Bowl XLV
2011: A Brief History Lesson
![Page 20: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/20.jpg)
I should use LXC…
![Page 21: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/21.jpg)
Checkbot (Codeship Classic)• Still running in production as our classic infrastructure
• Well-suited for users who want 1-click test environments without much customization
• Compromise flexibility for ease of use
![Page 22: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/22.jpg)
Checkbot
39K builds per day 7.8M builds per year
![Page 23: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/23.jpg)
Architecture
• Universal container with provided dependencies
• Fixed amount of available containers per VM
• Implement parallel testing pattern using pipelines with ParallelCI
• Users can have N pipelines running in isolation during a build
![Page 24: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/24.jpg)
User Commands
Universal Container
Pipeline
User Commands
Universal Container
Pipeline
Heroku
Deployment Provider
Capistrano
AppEngine
Elastic Beanstalk
etc…
![Page 25: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/25.jpg)
Limitations
• Parity between dev and test
• Can’t really debug locally
• No useable interface between user and container
• Resource consumption is too high
![Page 26: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/26.jpg)
We weren’t able to provide the best, most efficient product to our customers (or ourselves)
![Page 27: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/27.jpg)
Using Docker and the Docker Ecosystem
![Page 28: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/28.jpg)
Create a customizable, flexible test environment that enables us
to run tests in parallel
GOAL
![Page 29: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/29.jpg)
Big Wins with Docker
Even before 1.0, Docker was a clear choice
• Support and tooling • Standardization • Community of motivated developers
![Page 30: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/30.jpg)
Using Docker allowed us to build a much better testing platform than with LXC alone
![Page 31: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/31.jpg)
Codeship Jet
![Page 32: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/32.jpg)
A Docker-based Testing Platform
• Development started in 2014
• First beta in 2015
• Official launch February 2016
![Page 33: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/33.jpg)
A Docker-based Testing Platform
Built with Docker
in order to support Docker workflows
![Page 34: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/34.jpg)
Codeship Jet
2.3K builds per day ~250K total builds
![Page 35: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/35.jpg)
Docker Workflow Tools
• Docker Compose: service and step definition syntax
• Docker Registry: storage for images; remote caching*
• Docker for Mac and Windows: give users ability to reproduce CI environments locally
*not for long!
![Page 36: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/36.jpg)
![Page 37: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/37.jpg)
The workflow tools provided by Docker are indispensable
![Page 38: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/38.jpg)
Parallel Testing with Docker
![Page 39: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/39.jpg)
Managing containers with Docker allowed us to improve our
parallel testing workflow
![Page 40: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/40.jpg)
A New Parallel Workflow
• Introducing services adds additional layer of flexibility
• Loosen coupling between steps and services — execute N steps against M services
• Parallel and serial steps can be grouped and ordered in any way
![Page 41: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/41.jpg)
Services
• Pull image from any registry or build from Dockerfile
• Optimize service for testing tasks
• Fully customizable by the user
![Page 42: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/42.jpg)
Steps
• Each step is executed in an independent environment • Can be nested in serial and parallel groups • Two functions
• Run: execute a command against a service • Push: push image to registry
• Tag regex matching to run steps on certain branches or tagged releases
![Page 43: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/43.jpg)
User Commands
Universal Container
Pipeline
User Commands
Universal Container
Pipeline
User Commands
Universal Container
Pipeline
T1 T1 T1
![Page 44: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/44.jpg)
Step
po
stg
res
red
is
command
web
Step
po
stg
res
red
is
command
web
Step
command
ruby Step
po
stg
res
red
is
command
web
T1 T2 T3
![Page 45: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/45.jpg)
codeship-services.yml
db: image: postgres:9.5
app: encrypted_dockercfg_path: dockercfg.encrypted build: image: user/some-image dockerfile: Dockerfile.test cached: true links: - db deploy: encrypted_dockercfg_path: dockercfg.encrypted build: dockerfile: Dockerfile.deploy
![Page 46: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/46.jpg)
codeship-steps.yml
- type: serial steps: - type: parallel steps: - name: rspec service: app command: bin/ci spec - name: rubocop service: app command: rubocop - name: haml-lint service: app command: haml-lint app/views - name: rails_best_practices service: app command: bin/railsbp
- service: deploy type: push image_name: rheinwein/notes-app tag: ^master$ registry: https://index.docker.io/v1/ encrypted_dockercfg_path: dockercfg.encrypted
1
2
![Page 47: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/47.jpg)
ProTip: Your push or deploy step should never be part of a parallel step group
![Page 48: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/48.jpg)
Demo!
![Page 49: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/49.jpg)
Docker for Mac and Windows
• All users can test locally
• Jet CLI is available at bit.ly/codeship-jet-tool
• Free, and you don’t need a Codeship account
• HUGE advantage over our previous LXC implementation
![Page 50: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/50.jpg)
Engineering Challenges
![Page 51: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/51.jpg)
Infrastructure
Build allocation
• Customers can choose specs for their build machines
• Machine provisioning used to be part of the build process
• Now we pool build machines
• Allocation time is ~1 second!
![Page 52: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/52.jpg)
Performance
Image Caching
• Old way: rely on the registry for caching
• A pull gave us access to each parent layer; rebuilding the image used the local cache
• 1.10 content addressable breaking change
![Page 53: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/53.jpg)
Performance
Image Caching • Great news: 1.11 restores parent/child relationship
when you save the images via docker save • ETA: 1 month • Double-edged sword of relying on external tools ¯\_( )_/¯
![Page 54: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/54.jpg)
What’s Next?
![Page 55: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/55.jpg)
Swarm!
• Jet was born pre-Swarm
• We manage build machines on AWS via our own service
• Previous concerns about security — single tenancy
• Swarm (and services like Carina) are promising for the future
![Page 56: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/56.jpg)
libcompose
• Currently use APIs directly for container-level operations (Jet was also born before Fig was popular)
• Minimal change for our users and builds, but much easier for our engineers
• Preliminary work has been completed 🐳
![Page 57: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/57.jpg)
You can create a highly efficient parallel testing platform with LXC
alone, but using Docker tools makes it better
TL;DR
![Page 58: Efficient Parallel Testing with Docker](https://reader034.fdocuments.net/reader034/viewer/2022050613/5881642d1a28ab80508b66dd/html5/thumbnails/58.jpg)
Thank you!