DevOps With Smell
Transcript of DevOps With Smell
DEVOPS WITH SMELLANTONS KRANGA
~ WHOAMI
▸ Full stack developer ~ 15years
▸ Cloud Architect
▸ DevOps evangelist
▸ Innovation Center of Accenture Cloud Platform
▸ Speaker
▸ Marathon runner
ANTONS KRANGA
WHY DEVOPSDEVELOPERS VERSION
DRIVERS FOR SOFTWARE DEVELOPMENT
▸ development price
BEFORE AFTER
▸ innovations speed
DRIVERS FOR SOFTWARE DEVELOPMENT
▸ development price
BEFORE AFTER
▸ innovations speed
▸ specialization silos
▸ ops comes first
▸ cross functional
▸ software defined data centers
▸ complex communication ▸ complexity theory
DEVOPS IS AN ENVIRONMENT WHERE PEOPLE TALK
me :)
DEFINITION OF DEVOPS
SMELL SYMPTOMADEVOPS ANTIPATTERNS
ANTIPATTERN # 0
UNICORNS VS HORSES
You cannot be a unicorn when all others are horses
ANTIPATTERN # 0.1
DEVOPS IN A BOX
You cannot buy culture!
▸ Buy a Golden DevOps LVL5
▸ Become certified DevOps master
▸ Give me DevOps compliance checklist
▸ Give me 5 key DevOps control metrics
ANTIPATTERN # 0.2
REBRANDING
DevOps != Configuration ManagementDevOps != Release Management|DevOps != Product Management…DevOps != (*) Management
DEVOPS IS THE CULTURE YOU CANNOT FIND IN IN ITIL CATALOG
ANTIPATTERN # 1
DEVOPS AS A BLACK BOX
What is the value in DevOps?
! Developer
! Sys-Op
ANTIPATTERN # 2
SORRY, NOT MY DEPARTMENT
Bread ownership and specialization with autonomous teams
vs
http://martinfowler.com/bliki/DevOpsCulture.html
ANTIPATTERN # 2.1
DEVOPS AS A SILO
DevOps teams build their own Silo
▸ You cannot talk to DevOps
▸ Use Jira instead !!!!
▸ RTFM Architecture
ANTIPATTERN # 3
DEFINITION OF DONE
▸ DoD fails with “ilities”
▸ Services can be easily “undone”
ANTIPATTERN # 4
FEAR OF RELEASE
RELEASE === RISK
NO RELEAE NO REVENUE
vs
ANTIPATTERN # 5
SNOWFLAKE SERVER
Applying changes to Server Instance manually leads to unique and distinct server configuration footprint (TECHNICAL DEBT)
http://martinfowler.com/bliki/SnowflakeServer.html
VITAMINSDEVOPS ANTIPATTERNS
VITAMINS
INFRASTRUCTURE AS CODE
ENV
ENV
ENV
DEV …CODE
VITAMINS
INFRASTRUCTURE AS CODE
DEV
Code is aContract OPS
VITAMINS
INFRASTRUCTURE AS CODE
DEV
Code is aContract OPS
Put infrastructure into SCM
VITAMINS
INFRASTRUCTURE AS CODE
DEV
Code is aContract OPS`
Make infrastructure part of app baseline
VITAMINS
WE LIKE CI/CD
▸ We need feedback not to be afraid
▸ Feedback != SPAM
▸ It’s about size of release not frequency
▸ Don’t judge for broken builds
▸ Go home when build is green
PAINKILLERSDEVOPS ANTIPATTERNS
ANTIPATTERN # 6
“JENKINS” DRIVEN DEVELOPMENT
`
`
`
Pipelines are easily becomes your single point of failure
CI
ANTIPATTERN # 6: PROPOSED SOLUTION
SHIFTING RESPONSIBILITY
CI users (DEVs or OPs) are best for managing their jobs
▸ Use DSL to build CI/CD pipelines
▸ Store CI/CD pipelines in git if possible make it part of app baseline
▸ Generate and bootsrap pipelines with API
ANTIPATTERN # 7
SLOW PIPELINES
` ` `
Over-engineered Pipelines
Pipeline execution takes too long
Leads to delayed feedback
ANTIPATTERN # 6: PROPOSED SOLUTION
SLOW PIPELINES
`
`
`
Parallelize where you can!
ANTIPATTERN # 7
MANUAL PROMOTION
Engineer PRODUATIntegr TestsDEV System Tests
ENVENVENV ENV
ANTIPATTERN # 7: PROPOSED SOLUTION
KILL SWITCH FOR MANUAL TESTING
PRODUATIntegr TestsDEV System Tests
ENVENVENV ENV
TEST
ANTIPATTERN # 7: PROPOSED SOLUTION
AB TESTING
PROD A
Integr TestsDEV System Tests
ENV
ENVENV
TEST A
PROD B
ENV
TEST Bfeedback
feedback
measure
ANTIPATTERN # 7.1
CODEREVIEW
PRODUATIntegr Tests System Tests
ENVENVENV ENV
DEV B
feature branch DEV Ccode
review
ANTIPATTERN # 7.1: PROPOSED SOLUTION
REACTIVE CODEREVIEW
ReleaseIntegr Tests System Tests
ENVENV ENV
Full Regression
Calc technicaldebt
ENV
Nightly
DEV A
Standup
code review
…
ANTIPATTERN # 7.1
CODEREVIEW
PRODUATIntegr TestsDEV A System Tests
ENVENVENV ENVmaster
DEV B
feature branch DEV Ccode
review
ANTIPATTERN # 8
DEV DEPLOYMENT VIA CI
DEVENV
DEVENV
DEVENV
…DEV CI
ANTIPATTERN # 8: PROPOSED SOLUTION
DEV DEPLOYMENT VIA CI
DEV …NO CI
DEVENV
DEVENV
DEVENV
ANTIPATTERN # 8: PROPOSED SOLUTION
DEV DEPLOYMENT VIA CI
ENV
ENV
DEVENV
…DEV NO CI
WANT TO HACK? SURE!
ANTIPATTERN # 8: PROPOSED SOLUTION
DEV DEPLOYMENT VIA CI
ENV
ENV
DEVENV
…
YOU BROKE IT? YOU FIX IT!
DEV NO CI
ANTIBIOTICSDEVOPS ANTIPATTERNS
ANTIPATTERN # 9
GOLDEN IMAGE
VM
OS
Problems
▸ Maintained manually
▸ No collaboration
▸ Hard to distribute
▸ Non versioning
Chnorr Service
ANTIPATTERN # 9: PROPOSED SOLUTION
PROVISIONING
OS Chnorr Service
ANTIPATTERN # 9: PROPOSED SOLUTION
PROVISIONING
OS
Configure
Infrastructurecode
Harden Download Install
Chnorr Service
ANTIPATTERN # 9: PROPOSED SOLUTION
PROVISIONING
OS
Configure
Infrastructurecode
Harden Download Install
cmdb
Chnorr Service
ANTIPATTERN # A
FRAGILE PROVISIONING
OS
Configure
Infrastructurecode
Harden Download Install
▸ Provisioning success < 100%
▸ Time to provision
Chnorr Service
ANTIPATTERN # A
FRAGILE PROVISIONING
OS
Configure
Infrastructurecode
Harden Download Install
▸ Provisioning success < 100%
▸ Time to provision
Chnorr Service
ANTIPATTERN # A
FRAGILE PROVISIONING
OS
Configure
Infrastructurecode
Harden Download Install
▸ Provisioning success < 100%
▸ Time to provision
Chnorr Service
ANTIPATTERN # A
FRAGILE PROVISIONING
OS
Configure
Infrastructurecode
Harden Download Install
▸ Provisioning success < 100%
▸ Time to provision
Chnorr Service
ANTIPATTERN # A.1
PRIVATE DATA CENTER PROVISIONING
PRIVATE DATA CENTER: NO INTERNET!!!
OS
ConfigureHarden Download Install
Chnorr Service
ANTIPATTERN # A.1
PRIVATE DATA CENTER PROVISIONING
PRIVATE DATA CENTER: NO INTERNET!!!
OS
ConfigureHarden Download Install
…
apt yum gem
Chnorr Service
ANTIPATTERN # A.1
PRIVATE DATA CENTER PROVISIONING
PRIVATE DATA CENTER: NO INTERNET!!!
OS
ConfigureHarden Download Install
…
apt yum gem
complicated?
Chnorr Service
ANTIPATTERN # A.1: PROPOSED SOLUTION
STEM CELLS
static dynamic
OS
ConfigureHarden Download Install
Chnorr Service
ANTIPATTERN # A.1: PROPOSED SOLUTION
STEM CELLS
OS
ConfigureHarden Download Install
LAB PRIVATE DATA CENTER
Chnorr Service
ANTIPATTERN # A.1: PROPOSED SOLUTION
STEM CELLS
OS
ConfigureHarden Download Install
LAB PRIVATE DATA CENTER
code
packerPROVISIONSNAPSHOT
docker
Chnorr Service
ANTIPATTERN # B
DEPENDENCY HELL
OS
Configure
Infrastructurecode
Harden Download Install
▸ Version of libraries
▸ Version of packages
▸ Maintain dependencies
Chnorr Service
ANTIPATTERN # B
DEPENDENCY HELL
OS
Configure
Infrastructurecode
Harden Download Install
▸ Version of libraries
▸ Version of packages
▸ Maintain dependencies
▸ Version of your infra code
▸ Maintain dependencies
Chnorr Service
ANTIPATTERN # B: PROPOSED SOLUTION
CANARY BUILDS
Accept TestsCI
Unit Test Provision
…
PRECISE version libs
Accept TestsCI Unit Test Provision
…
LATEST version libs
Chnorr Service
Chnorr Service
ANTIPATTERN # C
INFRASTRUCTURE PETAttributes of Pet
▸ Have meaningful names
▸ Long living instance
▸ Often needs manual nursing
▸ Requires scary patching
▸ Leads to snowflakes
▸ PaaS is modern pet
ANTIPATTERN # C: PROPOSED SOLUTION
INFRASTRUCTURE CATTLEAttributes of Pet▸ Have numbers in its name
▸ Short living instance
▸ Immutable configuration
▸ Recreate instead of patching
▸ Requires careful planning
ANTIPATTERN # C: PROPOSED SOLUTION
PET TO TRANSFORMATION CATTLE EXAMPLE
Chnorr Service
Database
ANTIPATTERN # C: PROPOSED SOLUTION
PET TO TRANSFORMATION CATTLE EXAMPLE
Chnorr Service
Database
ANTIPATTERN # C: PROPOSED SOLUTION
PET TO TRANSFORMATION CATTLE EXAMPLE
Chnorr Service
Database
Cattle
Pet
ANTIPATTERN # C: PROPOSED SOLUTION
PET TO TRANSFORMATION CATTLE EXAMPLE
Chnorr Service
Database
Cattle
Pet
User Data
ANTIPATTERN # D
SECRETS LEAK
OS
Configure
Infrastructurecode
Install
cmdbwrong place
for your secretswrong place
for your secrets
Chnorr Service
ANTIPATTERN # D: PROPOSED SOLUTION
SECRETS LEAK
▸ Don’t store secrets with code
▸ Don’t store secrets with configuration
▸ Don’t leave secrets in service
Secrets DON’Ts
ANTIPATTERN # D: PROPOSED SOLUTION
SECRETS LEAK
▸ Secret can be leased and rotated
▸ tmpfs is your fiend
Secrets DOs
ANTIPATTERN # D: PROPOSED SOLUTION
SECURITY LEASING EXAMPLE
Chnorr Service vault
consul
IAM
Database
api
x hours leasing
AWS
TAKEAWAYSGOOD INFRA CODE
TAKEAWAYS
LEARN PATTERNS BEFORE TOOLS
Patterns Tools
vs
TAKEAWAYS
EVERYTHING MUST HAVE AN API
REST
DSL
CLI
TAKEAWAYS
SELF TESTABLE CODE
▸ Use assertions for infrastructure code
▸ Use acceptance test frameworks
Tests improves your confidence
TAKEAWAYS
READING
▸ Book: A Human Error Approach to Aviation Accident Analysis
▸ Author: Douglas A. Wiegmann Scott A. Shappell
▸ ISBN: 978-0754618737
TAKEAWAYS
READING
▸ Book: Clean Code
▸ Author: Robert C Martin
▸ ISBN: 978-0132350884
THANK YOU