Spot Trading - A case study in continuous delivery for mission critical financial services
-
Upload
saltstack -
Category
Technology
-
view
472 -
download
0
description
Transcript of Spot Trading - A case study in continuous delivery for mission critical financial services
![Page 1: Spot Trading - A case study in continuous delivery for mission critical financial services](https://reader033.fdocuments.net/reader033/viewer/2022061204/547e8af7b4795993508b4b9a/html5/thumbnails/1.jpg)
Automated deployments for mission-critical financial services
Jeremy Alons Systems Engineer, Spot Trading LLC Views and statements are my own and may not reflect those of my employer
v.8
![Page 2: Spot Trading - A case study in continuous delivery for mission critical financial services](https://reader033.fdocuments.net/reader033/viewer/2022061204/547e8af7b4795993508b4b9a/html5/thumbnails/2.jpg)
A case study in continuous delivery
![Page 3: Spot Trading - A case study in continuous delivery for mission critical financial services](https://reader033.fdocuments.net/reader033/viewer/2022061204/547e8af7b4795993508b4b9a/html5/thumbnails/3.jpg)
Why?
• Previously, deployments were time consuming
• Took hours of developer time, sometimes spread over weeks
• Manual process meant many problems at many points
![Page 4: Spot Trading - A case study in continuous delivery for mission critical financial services](https://reader033.fdocuments.net/reader033/viewer/2022061204/547e8af7b4795993508b4b9a/html5/thumbnails/4.jpg)
Our goals• Rapid feedback
• Tie configuration to deployment
• Be highly parallelized, if possible
• Infrastructure as code
• Repeatable
![Page 5: Spot Trading - A case study in continuous delivery for mission critical financial services](https://reader033.fdocuments.net/reader033/viewer/2022061204/547e8af7b4795993508b4b9a/html5/thumbnails/5.jpg)
Challenges
• Lots of unique software
• Differences force complexity
• Misused the tools we had originally
• Required additional tooling (that continues today)
![Page 6: Spot Trading - A case study in continuous delivery for mission critical financial services](https://reader033.fdocuments.net/reader033/viewer/2022061204/547e8af7b4795993508b4b9a/html5/thumbnails/6.jpg)
Steps: Build
Deploy Config
Run
![Page 7: Spot Trading - A case study in continuous delivery for mission critical financial services](https://reader033.fdocuments.net/reader033/viewer/2022061204/547e8af7b4795993508b4b9a/html5/thumbnails/7.jpg)
Steps: Build
Deploy!Config
Run
![Page 8: Spot Trading - A case study in continuous delivery for mission critical financial services](https://reader033.fdocuments.net/reader033/viewer/2022061204/547e8af7b4795993508b4b9a/html5/thumbnails/8.jpg)
Steps: Build
Deploy Config!
Run
![Page 9: Spot Trading - A case study in continuous delivery for mission critical financial services](https://reader033.fdocuments.net/reader033/viewer/2022061204/547e8af7b4795993508b4b9a/html5/thumbnails/9.jpg)
Build: Bamboo (or Jenkins, travis-ci, etc) executes commands
i.e. gcc -o foo foo.c sh tests.sh
etc
![Page 10: Spot Trading - A case study in continuous delivery for mission critical financial services](https://reader033.fdocuments.net/reader033/viewer/2022061204/547e8af7b4795993508b4b9a/html5/thumbnails/10.jpg)
Source code in. Artifacts out.
![Page 11: Spot Trading - A case study in continuous delivery for mission critical financial services](https://reader033.fdocuments.net/reader033/viewer/2022061204/547e8af7b4795993508b4b9a/html5/thumbnails/11.jpg)
Now what? We need to get artifacts to where they need to go
![Page 12: Spot Trading - A case study in continuous delivery for mission critical financial services](https://reader033.fdocuments.net/reader033/viewer/2022061204/547e8af7b4795993508b4b9a/html5/thumbnails/12.jpg)
This poses problems. • What hosts? • How do we get correct versions to those hosts? • Are we sure those hosts are suitable for the application?
![Page 13: Spot Trading - A case study in continuous delivery for mission critical financial services](https://reader033.fdocuments.net/reader033/viewer/2022061204/547e8af7b4795993508b4b9a/html5/thumbnails/13.jpg)
SaltStack to the rescue!
![Page 14: Spot Trading - A case study in continuous delivery for mission critical financial services](https://reader033.fdocuments.net/reader033/viewer/2022061204/547e8af7b4795993508b4b9a/html5/thumbnails/14.jpg)
Fast, scalable and flexible systems management software for data center
automation, cloud orchestration, server provisioning, configuration management
and more
- www.saltstack.com
![Page 15: Spot Trading - A case study in continuous delivery for mission critical financial services](https://reader033.fdocuments.net/reader033/viewer/2022061204/547e8af7b4795993508b4b9a/html5/thumbnails/15.jpg)
Fast, scalable and flexible systems management software for data center
automation, cloud orchestration, server provisioning, configuration management
and more
- www.saltstack.com
![Page 16: Spot Trading - A case study in continuous delivery for mission critical financial services](https://reader033.fdocuments.net/reader033/viewer/2022061204/547e8af7b4795993508b4b9a/html5/thumbnails/16.jpg)
Fast, scalable and flexible systems management software for data center
automation, cloud orchestration, server provisioning, configuration management
and more
- www.saltstack.com
![Page 17: Spot Trading - A case study in continuous delivery for mission critical financial services](https://reader033.fdocuments.net/reader033/viewer/2022061204/547e8af7b4795993508b4b9a/html5/thumbnails/17.jpg)
Fast, scalable and flexible systems management software for data center
automation, cloud orchestration, server provisioning, configuration management
and more
- www.saltstack.com
![Page 18: Spot Trading - A case study in continuous delivery for mission critical financial services](https://reader033.fdocuments.net/reader033/viewer/2022061204/547e8af7b4795993508b4b9a/html5/thumbnails/18.jpg)
Fast, scalable and flexible systems management software for data center
automation, cloud orchestration, server provisioning, configuration management
and more
- www.saltstack.com
![Page 19: Spot Trading - A case study in continuous delivery for mission critical financial services](https://reader033.fdocuments.net/reader033/viewer/2022061204/547e8af7b4795993508b4b9a/html5/thumbnails/19.jpg)
Salt is extremely extensible• External pillars
* Just python
• Interface with ZooKeeper for host discovery
• ZooKeeper maintains source of truth
• But how does Bamboo talk to ZooKeeper?
• Facilitate the communication through Salt
![Page 20: Spot Trading - A case study in continuous delivery for mission critical financial services](https://reader033.fdocuments.net/reader033/viewer/2022061204/547e8af7b4795993508b4b9a/html5/thumbnails/20.jpg)
Grains
• Static information about systems
• Loaded at minion start time
• OS (major, minor), ipv[4|6], etc
• Something you ask the server
Interface to derive information about the underlying system
![Page 21: Spot Trading - A case study in continuous delivery for mission critical financial services](https://reader033.fdocuments.net/reader033/viewer/2022061204/547e8af7b4795993508b4b9a/html5/thumbnails/21.jpg)
Pillar
• Something you tell a server
• “pillar” is a dictionary of items sent to minions
• ext_pillar are derived by the master by executing python
• Still sent to minions
Pillar is an interface for Salt designed to offer global values that can be distributed to all minions.
![Page 22: Spot Trading - A case study in continuous delivery for mission critical financial services](https://reader033.fdocuments.net/reader033/viewer/2022061204/547e8af7b4795993508b4b9a/html5/thumbnails/22.jpg)
Grain: Ask Pillar: Tell
![Page 23: Spot Trading - A case study in continuous delivery for mission critical financial services](https://reader033.fdocuments.net/reader033/viewer/2022061204/547e8af7b4795993508b4b9a/html5/thumbnails/23.jpg)
server1server2
Salt Master
ZooKeeper
Stash
Bamboo
Zoom
Build triggered
Selects build to deploy
Tells salt to find
matching daemons
Returns results
Runs deploy state
Asks forhosts
Return hosts to target
Displays real timestatus and
dependency map
Developer
Pushes commit1
23
4
5
6
7
8
0
![Page 24: Spot Trading - A case study in continuous delivery for mission critical financial services](https://reader033.fdocuments.net/reader033/viewer/2022061204/547e8af7b4795993508b4b9a/html5/thumbnails/24.jpg)
Bamboo talks to salt via Salt’s RESTful api
Salt talks to ZooKeeper via an external pillar
![Page 25: Spot Trading - A case study in continuous delivery for mission critical financial services](https://reader033.fdocuments.net/reader033/viewer/2022061204/547e8af7b4795993508b4b9a/html5/thumbnails/25.jpg)
Salt asks ZooKeeper “What hosts are currently responsible for app Foo?”
![Page 26: Spot Trading - A case study in continuous delivery for mission critical financial services](https://reader033.fdocuments.net/reader033/viewer/2022061204/547e8af7b4795993508b4b9a/html5/thumbnails/26.jpg)
ZooKeeper responds, we run a series of state files on the targeted servers
![Page 27: Spot Trading - A case study in continuous delivery for mission critical financial services](https://reader033.fdocuments.net/reader033/viewer/2022061204/547e8af7b4795993508b4b9a/html5/thumbnails/27.jpg)
States• Check stats (memory, disk, etc)
• - cmd.run salt://deploy/check_host.py
• Apply system settings (buffers, os monitoring, etc)
• Install software
• yum install Foo-release-2.3-10
![Page 28: Spot Trading - A case study in continuous delivery for mission critical financial services](https://reader033.fdocuments.net/reader033/viewer/2022061204/547e8af7b4795993508b4b9a/html5/thumbnails/28.jpg)
After build, package into rpm
• Use the tools available
• Versioning
• Well understood
• Reporting baked in
![Page 29: Spot Trading - A case study in continuous delivery for mission critical financial services](https://reader033.fdocuments.net/reader033/viewer/2022061204/547e8af7b4795993508b4b9a/html5/thumbnails/29.jpg)
RPM
• Specify version
• Allows control over latest build, n-1, etc
• Passed through via pillar at install time, fed by Bamboo deployment plan
• Allows Bamboo to drive version released
![Page 30: Spot Trading - A case study in continuous delivery for mission critical financial services](https://reader033.fdocuments.net/reader033/viewer/2022061204/547e8af7b4795993508b4b9a/html5/thumbnails/30.jpg)
API wrapper
• Home grown deployment software
• Command line utility that converts bamboo variables to salt states
• Deploys itself via the deployment process - Dog fooding the deployment process
![Page 31: Spot Trading - A case study in continuous delivery for mission critical financial services](https://reader033.fdocuments.net/reader033/viewer/2022061204/547e8af7b4795993508b4b9a/html5/thumbnails/31.jpg)
deploy state# Run the installation on the targeted server deploy_{{ application_name }}: pkg.installed: - name: {{ application_name }} - version: {{ pillar[‘deploy_pillar’][‘version’] }} - refresh: True - enablerepo: spot_applications - require: - sls: spot_applications_repo
![Page 32: Spot Trading - A case study in continuous delivery for mission critical financial services](https://reader033.fdocuments.net/reader033/viewer/2022061204/547e8af7b4795993508b4b9a/html5/thumbnails/32.jpg)
Things we’ve learned• In the end, it’s all just bits on disk
• All software is just software, and (should be) equal
• Be specific and exact
• Catch issues soon (test early, test often)
• Being clever will come back to haunt you (after all, this is just software development)
![Page 33: Spot Trading - A case study in continuous delivery for mission critical financial services](https://reader033.fdocuments.net/reader033/viewer/2022061204/547e8af7b4795993508b4b9a/html5/thumbnails/33.jpg)
Highlights
• Implemented deployment pipeline
• Artifact of the build
• Bringing components together
• Repeatable, rapid feedback
![Page 34: Spot Trading - A case study in continuous delivery for mission critical financial services](https://reader033.fdocuments.net/reader033/viewer/2022061204/547e8af7b4795993508b4b9a/html5/thumbnails/34.jpg)
Thank you.
Questions?
email: [email protected]
For more about some of the interesting work Spot is doing, check our Engineering Blog at
http://www.spottradingllc.com/technology/