Building and Deploying a Saas platform On Prem · CD Continuous Delivery vs Continuous Deployment...
Transcript of Building and Deploying a Saas platform On Prem · CD Continuous Delivery vs Continuous Deployment...
![Page 1: Building and Deploying a Saas platform On Prem · CD Continuous Delivery vs Continuous Deployment – “Continuous Delivery doesn't mean every change is deployed to production ASAP.](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed0b5f206b7b4202c4996d7/html5/thumbnails/1.jpg)
Building and Deploying a Saas platform On Prem
A Digital Asset Management System as a Service
Christophe Vanlancker@Carroarmato0
Slides by Michel van de Ven and
Julien Pivotto
![Page 2: Building and Deploying a Saas platform On Prem · CD Continuous Delivery vs Continuous Deployment – “Continuous Delivery doesn't mean every change is deployed to production ASAP.](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed0b5f206b7b4202c4996d7/html5/thumbnails/2.jpg)
Christophe VanlanckerChristophe Vanlancker
● Internal operations and consultingInternal operations and consulting● MentorMentor● Kris couldn’t make it so I ‘s/Kris/Christophe/g’Kris couldn’t make it so I ‘s/Kris/Christophe/g’
![Page 3: Building and Deploying a Saas platform On Prem · CD Continuous Delivery vs Continuous Deployment – “Continuous Delivery doesn't mean every change is deployed to production ASAP.](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed0b5f206b7b4202c4996d7/html5/thumbnails/3.jpg)
Inuits
● Inuits is an Open Source company– We contribute back
● +70 people in 4 countries (BE, NL, UA, CZ)● One language: English● We offer
– Consulting● Development● System Administration
– and a niche Saas Platform
![Page 4: Building and Deploying a Saas platform On Prem · CD Continuous Delivery vs Continuous Deployment – “Continuous Delivery doesn't mean every change is deployed to production ASAP.](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed0b5f206b7b4202c4996d7/html5/thumbnails/4.jpg)
MediaMosa
● Drupal-based Digital Asset Management system● Commissioned by SURFnet and Kennisnet● Open Source (GPLv2), Open Standards● Webservice oriented: REST● Store & retrieve assets● Manage metadata using open standards
– Dublin Core, Qualified DC, IEEE/LOM, CZP– OAI-PMH, BagIt
● Transcode video, audio, images, PDF, stream content● Users: Kennisnet, NIBG, UGent, UvA, TiU, RUG, Radboud, UOslo,
Avans, PolitieAcademie, Acquia (NBC Sport), Cineca● http://mediamosa.org
![Page 5: Building and Deploying a Saas platform On Prem · CD Continuous Delivery vs Continuous Deployment – “Continuous Delivery doesn't mean every change is deployed to production ASAP.](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed0b5f206b7b4202c4996d7/html5/thumbnails/5.jpg)
![Page 6: Building and Deploying a Saas platform On Prem · CD Continuous Delivery vs Continuous Deployment – “Continuous Delivery doesn't mean every change is deployed to production ASAP.](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed0b5f206b7b4202c4996d7/html5/thumbnails/6.jpg)
![Page 7: Building and Deploying a Saas platform On Prem · CD Continuous Delivery vs Continuous Deployment – “Continuous Delivery doesn't mean every change is deployed to production ASAP.](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed0b5f206b7b4202c4996d7/html5/thumbnails/7.jpg)
=
MediaMosa as a Service
![Page 8: Building and Deploying a Saas platform On Prem · CD Continuous Delivery vs Continuous Deployment – “Continuous Delivery doesn't mean every change is deployed to production ASAP.](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed0b5f206b7b4202c4996d7/html5/thumbnails/8.jpg)
MediaSalsa infrastructure (simplified)
● For each environment (DTAP)– Backends: Core service (MediaMosa)– Frontends: Optional– Web servers– Database servers– Solr servers– Transcoding servers
![Page 9: Building and Deploying a Saas platform On Prem · CD Continuous Delivery vs Continuous Deployment – “Continuous Delivery doesn't mean every change is deployed to production ASAP.](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed0b5f206b7b4202c4996d7/html5/thumbnails/9.jpg)
Ideal world vs budget and reality→ pragmatic approach
![Page 10: Building and Deploying a Saas platform On Prem · CD Continuous Delivery vs Continuous Deployment – “Continuous Delivery doesn't mean every change is deployed to production ASAP.](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed0b5f206b7b4202c4996d7/html5/thumbnails/10.jpg)
CultureAutomation
MeasurementSharing
![Page 11: Building and Deploying a Saas platform On Prem · CD Continuous Delivery vs Continuous Deployment – “Continuous Delivery doesn't mean every change is deployed to production ASAP.](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed0b5f206b7b4202c4996d7/html5/thumbnails/11.jpg)
Puppet
Puppet automates all the things
→ mcollective orchestrates all the things
![Page 12: Building and Deploying a Saas platform On Prem · CD Continuous Delivery vs Continuous Deployment – “Continuous Delivery doesn't mean every change is deployed to production ASAP.](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed0b5f206b7b4202c4996d7/html5/thumbnails/12.jpg)
CD
● Continuous Delivery vs Continuous Deployment– “Continuous Delivery doesn't mean every change is deployed to production
ASAP. It means every change is proven to be deployable at any time” (@ccaum)
● Puppet code– Deployed to dev environment– Same puppet code for each environment– User-triggered deployments to UAT & Prod– Feature flags in Puppet code per environment (switchable architecture)
● Application code– Continuous integration in dev– User-triggered deployments to UAT – Deployment to prod is a business decision
![Page 13: Building and Deploying a Saas platform On Prem · CD Continuous Delivery vs Continuous Deployment – “Continuous Delivery doesn't mean every change is deployed to production ASAP.](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed0b5f206b7b4202c4996d7/html5/thumbnails/13.jpg)
Testing
● Developers test a lot, but– The tests don’t work– It works on my machine™– Wrong platform– Wrong PHP version
Fixed now, thanks to Jenkins!
and Vagrant...
![Page 14: Building and Deploying a Saas platform On Prem · CD Continuous Delivery vs Continuous Deployment – “Continuous Delivery doesn't mean every change is deployed to production ASAP.](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed0b5f206b7b4202c4996d7/html5/thumbnails/14.jpg)
Version Control
● Git● Code is under revision control
– Prefer small commits– Local features branches
● Infrastructure as code → git / hiera
![Page 15: Building and Deploying a Saas platform On Prem · CD Continuous Delivery vs Continuous Deployment – “Continuous Delivery doesn't mean every change is deployed to production ASAP.](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed0b5f206b7b4202c4996d7/html5/thumbnails/15.jpg)
Using OS packaging system
● Consistency, security, dependencies● Uniquely identify where files are coming from● Source repo may not be reachable● Little overhead when you automate● Configuration does not belong in a package
![Page 16: Building and Deploying a Saas platform On Prem · CD Continuous Delivery vs Continuous Deployment – “Continuous Delivery doesn't mean every change is deployed to production ASAP.](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed0b5f206b7b4202c4996d7/html5/thumbnails/16.jpg)
Pipelines
● A collection of jobs● Run in sequence● Start on checkout, end on deployment● From the developers’ side:
→ Git push
← Mail with changes + link to deploy
![Page 17: Building and Deploying a Saas platform On Prem · CD Continuous Delivery vs Continuous Deployment – “Continuous Delivery doesn't mean every change is deployed to production ASAP.](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed0b5f206b7b4202c4996d7/html5/thumbnails/17.jpg)
Pipelines steps
● Checkout● Syntax: php -l● Style: Drupal Coder● Package: FPM● Deploy to dev environment: mcollective● Tests in dev environment: drush run-tests● Publish package and promote: mcollective
![Page 18: Building and Deploying a Saas platform On Prem · CD Continuous Delivery vs Continuous Deployment – “Continuous Delivery doesn't mean every change is deployed to production ASAP.](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed0b5f206b7b4202c4996d7/html5/thumbnails/18.jpg)
Same Pipelines, Tools, Patterns are used by both devs and ops
![Page 19: Building and Deploying a Saas platform On Prem · CD Continuous Delivery vs Continuous Deployment – “Continuous Delivery doesn't mean every change is deployed to production ASAP.](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed0b5f206b7b4202c4996d7/html5/thumbnails/19.jpg)
CultureAutomation
MeasurementSharing
![Page 20: Building and Deploying a Saas platform On Prem · CD Continuous Delivery vs Continuous Deployment – “Continuous Delivery doesn't mean every change is deployed to production ASAP.](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed0b5f206b7b4202c4996d7/html5/thumbnails/20.jpg)
Logstash
● Collect all the logs– Drupal logs– Apache logs– Deployment logs– System logs
● Interpret, filter and correlate them● Logstash, ElasticSearch, Kibana, statsd, Graphite
![Page 21: Building and Deploying a Saas platform On Prem · CD Continuous Delivery vs Continuous Deployment – “Continuous Delivery doesn't mean every change is deployed to production ASAP.](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed0b5f206b7b4202c4996d7/html5/thumbnails/21.jpg)
Icinga
● Monitor everything– vhosts– databases– cronjobs– unit test suites
![Page 22: Building and Deploying a Saas platform On Prem · CD Continuous Delivery vs Continuous Deployment – “Continuous Delivery doesn't mean every change is deployed to production ASAP.](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed0b5f206b7b4202c4996d7/html5/thumbnails/22.jpg)
Graphite + gdash
● Collectd● Monitor platform usage● FFmpeg usage● Number of accounts ● Pipelined !
![Page 23: Building and Deploying a Saas platform On Prem · CD Continuous Delivery vs Continuous Deployment – “Continuous Delivery doesn't mean every change is deployed to production ASAP.](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed0b5f206b7b4202c4996d7/html5/thumbnails/23.jpg)
CultureAutomation
MeasurementSharing
![Page 24: Building and Deploying a Saas platform On Prem · CD Continuous Delivery vs Continuous Deployment – “Continuous Delivery doesn't mean every change is deployed to production ASAP.](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed0b5f206b7b4202c4996d7/html5/thumbnails/24.jpg)
Open Source
● Mediamosa is Fully Open Source● Lots of the PuppetCode to deploy it● Our passwords etc aren't
![Page 25: Building and Deploying a Saas platform On Prem · CD Continuous Delivery vs Continuous Deployment – “Continuous Delivery doesn't mean every change is deployed to production ASAP.](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed0b5f206b7b4202c4996d7/html5/thumbnails/25.jpg)
MediaSalsa Deployments
● Initially – 1 instance Academic usage @SurfNet– 1 Instance Commercial DC for non-edu
● Today– 2 academic instances– 1 commercial Saas instance– 2 on prem deployments
![Page 26: Building and Deploying a Saas platform On Prem · CD Continuous Delivery vs Continuous Deployment – “Continuous Delivery doesn't mean every change is deployed to production ASAP.](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed0b5f206b7b4202c4996d7/html5/thumbnails/26.jpg)
Why multiple Deployments
● “Security”– Academic Customer wanted a private tenant for security and
privacy● Initial hardware investment done already
– Public Tender , $customer bought huge amount of storage– Saas solution charges per TB– Asked for custom manual deployment
● CIO’s don’t believe in Cloud/SAAS (2017 !!!!)
![Page 27: Building and Deploying a Saas platform On Prem · CD Continuous Delivery vs Continuous Deployment – “Continuous Delivery doesn't mean every change is deployed to production ASAP.](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed0b5f206b7b4202c4996d7/html5/thumbnails/27.jpg)
Saas vs OnPrem
We have automated everything,
Infrastructure as Code , Pipeline as Code, Continuous delivery , so deploying this stack another time should be trivial !!
![Page 28: Building and Deploying a Saas platform On Prem · CD Continuous Delivery vs Continuous Deployment – “Continuous Delivery doesn't mean every change is deployed to production ASAP.](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed0b5f206b7b4202c4996d7/html5/thumbnails/28.jpg)
WRONG
![Page 29: Building and Deploying a Saas platform On Prem · CD Continuous Delivery vs Continuous Deployment – “Continuous Delivery doesn't mean every change is deployed to production ASAP.](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed0b5f206b7b4202c4996d7/html5/thumbnails/29.jpg)
Biased Automation
● Works in our infra , our constraints, our expectations● We expect to have access to our infra
● Puppet, monitoring, metrics, repos , jenkins
![Page 30: Building and Deploying a Saas platform On Prem · CD Continuous Delivery vs Continuous Deployment – “Continuous Delivery doesn't mean every change is deployed to production ASAP.](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed0b5f206b7b4202c4996d7/html5/thumbnails/30.jpg)
VM Provisioning
● Different Technologies – Open vs Proprietary– Guess which one is more problematic
● No access to Internal repositories● Network topologies● Having to ask to reboot a host● Having to ask to grow a VM
![Page 31: Building and Deploying a Saas platform On Prem · CD Continuous Delivery vs Continuous Deployment – “Continuous Delivery doesn't mean every change is deployed to production ASAP.](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed0b5f206b7b4202c4996d7/html5/thumbnails/31.jpg)
Security
● IPSec links to all stacks– Our own network complexity has grown exponentially
● Our network = Trusted ● Their network = Hostile
– Different approach in host vs network based firewalling
● User management – Only our accounts in our stack , our ldap– They want accounts
![Page 32: Building and Deploying a Saas platform On Prem · CD Continuous Delivery vs Continuous Deployment – “Continuous Delivery doesn't mean every change is deployed to production ASAP.](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed0b5f206b7b4202c4996d7/html5/thumbnails/32.jpg)
Variants
● We don’t want exceptions● They do want exceptions
● Old purchasing mentality– Custom Features– Additional Components– It’s “Their” stack
● Exceptions need to be codified in our infra
![Page 33: Building and Deploying a Saas platform On Prem · CD Continuous Delivery vs Continuous Deployment – “Continuous Delivery doesn't mean every change is deployed to production ASAP.](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed0b5f206b7b4202c4996d7/html5/thumbnails/33.jpg)
Continuous Deployment Delivery
● Deployment isn’t our decision anymore● Back to fixed deployment windows :(● Coordination with $customer on when to deploy● Even for Security Fixes
● For every single instance
![Page 34: Building and Deploying a Saas platform On Prem · CD Continuous Delivery vs Continuous Deployment – “Continuous Delivery doesn't mean every change is deployed to production ASAP.](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed0b5f206b7b4202c4996d7/html5/thumbnails/34.jpg)
Ideal world vs budget and reality→ pragmatic approach
![Page 35: Building and Deploying a Saas platform On Prem · CD Continuous Delivery vs Continuous Deployment – “Continuous Delivery doesn't mean every change is deployed to production ASAP.](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed0b5f206b7b4202c4996d7/html5/thumbnails/35.jpg)
Extreme Cost Difference
● The effort to run 5 stacks in your own infrastructure within your team is smaller than running 1 additional stack on prem at a customer
● Your pragmatic approach does not fit their infrastructure
● You will need to implement features (security/ storage support) that you do not need for your SAAS platform.
![Page 36: Building and Deploying a Saas platform On Prem · CD Continuous Delivery vs Continuous Deployment – “Continuous Delivery doesn't mean every change is deployed to production ASAP.](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed0b5f206b7b4202c4996d7/html5/thumbnails/36.jpg)
It could have been worse
● We are an Open Source company● All of our Choices are Open Source by default
– We could deploy full stacks On Prem– Including metrics, log analytics and monitoring– We had no external dependencies– No additional license costs