Continuous integration is not a solved problem
-
Upload
kristian-van-der-vliet -
Category
Software
-
view
144 -
download
1
Transcript of Continuous integration is not a solved problem
![Page 1: Continuous integration is not a solved problem](https://reader031.fdocuments.net/reader031/viewer/2022030313/58a3732f1a28abaa488b4a35/html5/thumbnails/1.jpg)
https://cyclid.io @cyclidci github.com/Cyclid/
Continuous Integration is not a solved problem
Config Management Camp, Gent 2017
![Page 2: Continuous integration is not a solved problem](https://reader031.fdocuments.net/reader031/viewer/2022030313/58a3732f1a28abaa488b4a35/html5/thumbnails/2.jpg)
https://cyclid.io @cyclidci github.com/Cyclid/
Kristian van der Vliet
● DevOpsing hard for the previous 7+ yearsPuppet 0.24Chef 0.9
● Gainfully employed as an SRE for a CDN company
● You may remember me from such projects as Syllable
● Don’t be fooled by the name, mijn Nederlands is niet goed!
![Page 3: Continuous integration is not a solved problem](https://reader031.fdocuments.net/reader031/viewer/2022030313/58a3732f1a28abaa488b4a35/html5/thumbnails/3.jpg)
https://cyclid.io @cyclidci github.com/Cyclid/
Why isn’t CI a solved problem?
● I’ve seen many CI pipelines for things like Chef; I’ve never met one I liked● Continuous Integration tools make assumptions
○ You’re “building” software○ It will result in an “artifact”○ Your software is one of a known set of languages○ The build process is well defined
● None of these are true for “non-traditional” software○ Configuration Management/Infrastructure as Code○ Network configuration○ Database schemas
![Page 4: Continuous integration is not a solved problem](https://reader031.fdocuments.net/reader031/viewer/2022030313/58a3732f1a28abaa488b4a35/html5/thumbnails/4.jpg)
https://cyclid.io @cyclidci github.com/Cyclid/
![Page 5: Continuous integration is not a solved problem](https://reader031.fdocuments.net/reader031/viewer/2022030313/58a3732f1a28abaa488b4a35/html5/thumbnails/5.jpg)
https://cyclid.io @cyclidci github.com/Cyclid/
“How hard can it be?”
![Page 6: Continuous integration is not a solved problem](https://reader031.fdocuments.net/reader031/viewer/2022030313/58a3732f1a28abaa488b4a35/html5/thumbnails/6.jpg)
https://cyclid.io @cyclidci github.com/Cyclid/
My ideal Continuous Integration
● Unopinionated Not tied to any workflow or toolset.
● Agnostic Tools change constantly.
● Flexible ...so do functional requirements.
● Operable I’m an Operations person!
● Open Source I’m also an Open Source person.
![Page 7: Continuous integration is not a solved problem](https://reader031.fdocuments.net/reader031/viewer/2022030313/58a3732f1a28abaa488b4a35/html5/thumbnails/7.jpg)
https://cyclid.io @cyclidci github.com/Cyclid/
![Page 8: Continuous integration is not a solved problem](https://reader031.fdocuments.net/reader031/viewer/2022030313/58a3732f1a28abaa488b4a35/html5/thumbnails/8.jpg)
https://cyclid.io @cyclidci github.com/Cyclid/
Unopinionated? Agnostic? Flexible? Operable? Open Source?
Jenkins Y Y Y N Y
Travis N N N N Note 1
CircleCI N N N - N
Codeship Y N N - N
GoCD Y N Y Note 2 Y
Semaphore Y N N - N
Concourse Y N Note 3 Note 4 Y
Drone Note 5 N N Y Y
1. Mostly Open Source but key components are not.2. YUM/APT packages, but UI or giant XML configuration file of Doom configuration.3. Concourse supports plugins but the plugin interface is basic.4. Basic setup is simple (Go binary) but it gets complex, fast E.g. BOSH clusters. Lots of components.5. Drone jobs are single scripts, so yes in the sense that you can anything you can do in a script.
![Page 9: Continuous integration is not a solved problem](https://reader031.fdocuments.net/reader031/viewer/2022030313/58a3732f1a28abaa488b4a35/html5/thumbnails/9.jpg)
https://cyclid.io @cyclidci github.com/Cyclid/
![Page 10: Continuous integration is not a solved problem](https://reader031.fdocuments.net/reader031/viewer/2022030313/58a3732f1a28abaa488b4a35/html5/thumbnails/10.jpg)
https://cyclid.io @cyclidci github.com/Cyclid/
● Written in Ruby, REST API○ Boring technologies, because boring means well understood○ I wanted plugins!
● “Everything is a plugin” philosophy○ Even where something isn’t a plugin, it’s highly decoupled
● Keep the component & dependency footprint as small as possible○ Cyclid, Nginx or Apache, Sidekiq, Redis, “a SQL database”
![Page 11: Continuous integration is not a solved problem](https://reader031.fdocuments.net/reader031/viewer/2022030313/58a3732f1a28abaa488b4a35/html5/thumbnails/11.jpg)
https://cyclid.io @cyclidci github.com/Cyclid/
Everything is a plugin
● Builder Create build instances to run jobs● Transport Communicate with build instances● Provisioner Configures build instances● Action Do things when running a job● Source Get the source(s) to build● API Extend the core REST API● Dispatcher Run a job somewhere
![Page 12: Continuous integration is not a solved problem](https://reader031.fdocuments.net/reader031/viewer/2022030313/58a3732f1a28abaa488b4a35/html5/thumbnails/12.jpg)
https://cyclid.io @cyclidci github.com/Cyclid/
Builders● Google● Digitalocean● LXD
Provisioners● Ubuntu● Debian
Transports● SSH● LXD
Actions● Command● Script● Log● Slack● Email
Sources● Git
API● Github
Dispatchers● “Local” (Sidekiq)
![Page 13: Continuous integration is not a solved problem](https://reader031.fdocuments.net/reader031/viewer/2022030313/58a3732f1a28abaa488b4a35/html5/thumbnails/13.jpg)
https://cyclid.io @cyclidci github.com/Cyclid/
{ "name" : "minimal-example", "environment": {
}, "stages" : [ { "name" : "hello-world", "steps" : [ { "action" : "log", "message" : "hello from %{name}" } ] } ], "sequence" : [ { "stage" : "hello-world" } ]}
● Jobs can be JSON or YAML
● Single file, with the source
● Defines the Environment,
Stages that make up the
Sequence and the Sequence
itself (pipeline)
![Page 14: Continuous integration is not a solved problem](https://reader031.fdocuments.net/reader031/viewer/2022030313/58a3732f1a28abaa488b4a35/html5/thumbnails/14.jpg)
https://cyclid.io @cyclidci github.com/Cyclid/
---name: minimal-exampleenvironment: {}stages:- name: hello-world steps: - action: log message: Hello from %{name}sequence:- stage: hello-world
![Page 15: Continuous integration is not a solved problem](https://reader031.fdocuments.net/reader031/viewer/2022030313/58a3732f1a28abaa488b4a35/html5/thumbnails/15.jpg)
https://cyclid.io @cyclidci github.com/Cyclid/
2017-01-07 12:49:58 +0000 : Obtaining build host...2017-01-07 12:51:38 +0000 : Preparing build host...==============================================================================================================================================================2017-01-07 12:51:38 +0000 : Job started. Context: {"job_id"=>88, "job_name"=>"minimal-example", "job_version"=>"1.0.0", "organization"=>"test", "name"=>"buildhost-a2713b62c93f7fd828c35261b1535614", "host"=>"104.197.96.230", "username"=>"build", "workspace"=>"/home/build", "password"=>nil, "key"=>"/etc/mist/id_rsa_build", "server"=>"prod-euw-build01", "distro"=>"ubuntu", "release"=>"trusty"}-------------------------------------------------------------------------------2017-01-07 12:51:39 +0000 : Running stage hello-world v1.0.0Hello from buildhost-a2713b62c93f7fd828c35261b1535614
![Page 16: Continuous integration is not a solved problem](https://reader031.fdocuments.net/reader031/viewer/2022030313/58a3732f1a28abaa488b4a35/html5/thumbnails/16.jpg)
https://cyclid.io @cyclidci github.com/Cyclid/
- name: bundle-install steps: - action: command cmd: sudo gem install bundler --no-ri --no-doc - action: command cmd: bundle install --path vendor/bundle path: '%{workspace}/Cyclid'
sequence:- stage: bundle-install on_success: lint on_failure: failure- stage: lint on_success: rspec on_failure: failure...
![Page 17: Continuous integration is not a solved problem](https://reader031.fdocuments.net/reader031/viewer/2022030313/58a3732f1a28abaa488b4a35/html5/thumbnails/17.jpg)
https://cyclid.io @cyclidci github.com/Cyclid/
Even more jobs
● Environments○ Operating System/Distro & version○ Additional package repositories○ Additional packages
● Sources● Secrets
○ Tokens, passwords, keys etc.
● Stages can also be defined on the server
![Page 18: Continuous integration is not a solved problem](https://reader031.fdocuments.net/reader031/viewer/2022030313/58a3732f1a28abaa488b4a35/html5/thumbnails/18.jpg)
https://cyclid.io @cyclidci github.com/Cyclid/
$ cyclid org showName: testOwner Email: [email protected] Key: -----BEGIN PUBLIC KEY-----...-----END PUBLIC KEY-----Members:
vanders$ cyclid job submit job.yamlJob: 89$ cyclid job status 89Status: Waiting$ cyclid job log 892017-01-07 13:38:04 +0000 : Obtaining build host......
![Page 19: Continuous integration is not a solved problem](https://reader031.fdocuments.net/reader031/viewer/2022030313/58a3732f1a28abaa488b4a35/html5/thumbnails/19.jpg)
https://cyclid.io @cyclidci github.com/Cyclid/
![Page 20: Continuous integration is not a solved problem](https://reader031.fdocuments.net/reader031/viewer/2022030313/58a3732f1a28abaa488b4a35/html5/thumbnails/20.jpg)
https://cyclid.io @cyclidci github.com/Cyclid/
The future!
![Page 21: Continuous integration is not a solved problem](https://reader031.fdocuments.net/reader031/viewer/2022030313/58a3732f1a28abaa488b4a35/html5/thumbnails/21.jpg)
https://cyclid.io @cyclidci github.com/Cyclid/
What we’ve learned from Config Mgmt
1. Flexibility is good2. It’s fine to have more than one tool
Competition is good3. Different tools and workflows suit different users4. Composability is good5. Data driven is good
DRY
![Page 22: Continuous integration is not a solved problem](https://reader031.fdocuments.net/reader031/viewer/2022030313/58a3732f1a28abaa488b4a35/html5/thumbnails/22.jpg)
https://cyclid.io @cyclidci github.com/Cyclid/
Composable CI
● Every CI job is a snowflake● Cyclid is built with Composability in mind
○ Every Job & Stage is versioned○ Stages can be defined on the server and used across different jobs
● Next: shareable Stages○ We’ve already done it with Config Management (Forge, Supermarket, Modules
etc.)○ Just another server with an API○ ...but will require dependency management
![Page 23: Continuous integration is not a solved problem](https://reader031.fdocuments.net/reader031/viewer/2022030313/58a3732f1a28abaa488b4a35/html5/thumbnails/23.jpg)
https://cyclid.io @cyclidci github.com/Cyclid/
Data driven
● Data Driven is a natural requirement of Composability● What would it even look for for Continuous Integration?
○ What data sources would be useful?○ How do we organize that data?○ How do we expose that data to both jobs & users?
● Jenkins is the closest with Parameterized Builds○ Limited in scope
● I genuinely don’t know what this looks like○ Let’s start the conversation!
![Page 24: Continuous integration is not a solved problem](https://reader031.fdocuments.net/reader031/viewer/2022030313/58a3732f1a28abaa488b4a35/html5/thumbnails/24.jpg)
https://cyclid.io @cyclidci github.com/Cyclid/
Logic
● Let’s stop pretending CI jobs don’t require logic○ Stop externalizing the logic into shell scripts
● only_if/not_if○ Same as conditionals on resources in Configuration Management tools
sequence:- stage: release only_if: %{branch} eq ‘master’
![Page 25: Continuous integration is not a solved problem](https://reader031.fdocuments.net/reader031/viewer/2022030313/58a3732f1a28abaa488b4a35/html5/thumbnails/25.jpg)
https://cyclid.io @cyclidci github.com/Cyclid/
Event Framework
● Cyclid jobs are run by a simple state machine● Originally I tried to make “Notifiers” Action plugins
○ Whoops!
● Events will be emitted during the job run, and consumed by plugins etc.○ Fan-out & Fan-in (Pipelines!)○ Proper Notifications, and Notification chains○ Deep job introspection & profiling○ Monitoring & Statistics
![Page 26: Continuous integration is not a solved problem](https://reader031.fdocuments.net/reader031/viewer/2022030313/58a3732f1a28abaa488b4a35/html5/thumbnails/26.jpg)
https://cyclid.io @cyclidci github.com/Cyclid/
Lots more
● More Plugins○ AWS Builder○ Redhat-ish Provisioner(s)○ Not-Linux○ Deployment
● Docker?● Vagrant?● Qemu?
![Page 27: Continuous integration is not a solved problem](https://reader031.fdocuments.net/reader031/viewer/2022030313/58a3732f1a28abaa488b4a35/html5/thumbnails/27.jpg)
https://cyclid.io @cyclidci github.com/Cyclid/
Questions?
![Page 28: Continuous integration is not a solved problem](https://reader031.fdocuments.net/reader031/viewer/2022030313/58a3732f1a28abaa488b4a35/html5/thumbnails/28.jpg)
https://cyclid.io @cyclidci github.com/Cyclid/
Thank you!
Website cyclid.io
Twitter @cyclidci
Github github.com/cyclid
Email [email protected]@cyclid.io