SaltConf14 - Eric johnson, Google - Orchestrating Google Compute Engine with SaltStack
SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps
-
Upload
saltstack -
Category
Technology
-
view
428 -
download
0
description
Transcript of SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps
Mentoring Devs Into DevOpsJustin Carmony Director of Development Deseret Digital Media
Lets Measure The Audience
• Who here is a…
• System Administrator?
• Developer?
• Manager / Management?
• “DevOp?”
Confession:I’m a Developer
Self-Taught OpsBecause There Was No One Else To Do It
About Me• Director of Development
for Deseret Digital Media
• Utah PHP UsergroupPresident
• I Make (and Break) Web Stuff (10 years)
• Salt User in Production since 0.8
(I <3 Salt)
This Presentation• Lessons learned at DDM & previous jobs
• Insight into our process of increasing “DevOps”
• We’re still learning, but this what we’ve found.
• Slides will be posted online, so don’t worry about copying slide content.
• Feel free to ask on-topic questions during, and we’ll have questions at the end.
About DDM
• Deseret Digital Media runs local website like KSL.com, DeseretNews.com
• Running National and International Websites like OK.com, familia.com.br, etc.
• ~10 million pageviews a day across sites.
• ~150 internal VMs, a few dozen physical machines, some AWS sprinkled around.
Lets Start With a Story!
You Work for an Awesome Tech Company
Team Is Working Hard to Build New Things!
You launch your awesome product!
A Few More Features…
… and next thing you know…
Awesome Job Team, We Rock!
We Need !Real-Time XYZ Feature!
ASAP!
We Need !Real-Time XYZ Feature!
ASAP!
We Need !Real-Time XYZ Feature!
ASAP!
&#$%!
“Huh, it works if you ! just turn off caching…”!
- Dev @ 80th Hour This Week
“I’m sure this ! will work…”
“Our servers are melting!”
“We Need a Better Solution!”
So…
Where Do We Start?
We Have This Problem
Challenges We Faced
• Giant mesh-up of technologies
• Tightly-coupled & fragile infrastructure
• Debugging production only bugs was difficult
• Bugs that were part code, part environment were a nightmare to track down.
So One Day… We Had A Genius Idea!
Lets Hire a DevOp!
I’m Not JokingWe Actually Said This
Two Problems with this “Idea”
Problem #1 - We Didn’t Understand What We Really Wanted
Step 1: Hire a DevOp!
Step 2: ????????????!
Step 3: Profit!
Step 1: Hire a DevOp!
Step 2: ????????????!
Step 3: Profit! Everything Works !Perfectly!
Problem #2 - People Who Are Great At Dev & Ops Are Hard To Find
Expectation:
Reality:
Honest Team Discussion: What is it we’re really looking for?
We Discovered a Few Things
What does DevOps Mean To Us?
• DevOps: Dev & Ops, a Culture of Collaboration
• Our Goal: “10 deploys a day without issues”
• Everyone shares the goal of quick development of features AND a stable system that stays up.
Team StructureDevs: 30 Ops: 2
Team StructureDevs: 30 Ops: 2
DevOps: 1
Team StructureDevs: 30 Ops: 2
DevOps: 1
Team StructureDevs: 30 Ops: 2
DevOps: 1
Hiring one person won’t just solve all our problems!
Team Realizations
• Hardest problem already solved: awesome team
• No foreseeable rapid expansion, must operate at our current scale
• Each Project’s Director of Development was acting as the bridge between Dev and Ops, but would become a bottleneck.
Teams Already Had Some Ad-Hoc DevOps Tools
- Real-time Logging - Capistrano Deploys
- Nagios Alerts - Server Metrics - Puppet for File Mgmt
- App Stats w/ Graphite - Graphite Dashboards - Salt for Cfg Management
- Homebrewed Metrics Sys. - Homebrewed Alert System
Teams Already Had Some Ad-Hoc DevOps Tools
- Real-time Logging - Capistrano Deploys
- Nagios Alerts - Server Metrics - Puppet for File Mgmt
- App Stats w/ Graphite - Graphite Dashboards - Salt for Cfg Management
- Homebrewed Metrics Sys. - Homebrewed Alert System
Step 1: Hire a DevOp!
Step 2: ????????????!
Step 3: Profit! Everything Works !Perfectly!
Step 1: Hire a DevOp!
Step 2: ????????????!
Step 3: Profit! Everything Works !Perfectly!
We Formed A Strategy
Step #1: Promote Dev to DevOp Role
WAIT!Isn’t that the advice you just
said was a bad idea?!
DevOp Engineer
• Well Defined Role:
• Ownership over the TOOLS to improve DevOps efforts.
• Resource for other teams to help use DevOps Tools.
• Easy to work with, aptitude for systems & ops, likes to try new things.
Promoting From Within
• A seasoned dev for your team already knows:
• Your Pain Points
• Your System’s Quirks
• How the “Chaos Works”
• Knows the people & personalities on your team
Step #2: Change Team Structure
Team StructureDevs: 30 Ops: 2
Team StructureDevs: 30 Ops: 2
Team StructureGoal: Spread Out Expertise By Increasing
Ops Experience & Skills Among Devs
Dev Ops
Team StructureGoal: Spread Out Expertise By Increasing
Ops Experience & Skills Among Devs
Dev Ops
Team Structure
Dev Ops
Team Structure
Dev Ops
Increasing Ops Among Devs
• Identify Devs who liked “Ops” & wanted to Learn
• Pair Dev with Op / Director
• Learning Dev works on things, not Op /Director.
• Pair program if needed.
Step #3: Increase Everyone’s Insight
Step #3: Increase Everyone’s Insight
Metrics• Everyone has access to Network, Server, and
Application Metrics.
• Consolidate & reduce places to look. We try to pipe everything to StatsD / Graphite
• Each developer trained to add & track metrics in production.
• We’re okay with 98% uptime of stats to avoid complexity.
Real-Time Logging
Real-Time Logging• Harder & more complicated at scale
• Still trying to solve well, we have lots of logs.
• Start with small window of data (i.e. 48 hours) and start to expand window.
• We’re trying Logstash, ElasticSearch, and Kibana right now.
• Generate Statistics off our Logs
Tracking Changes
• Everything, everything, everything in git (we use GitHub)
• Everyone has access to all repos • Everyone does work through Pull Requests • Everyone has their work code reviewed *
* - Your can merge w/o a review, but must be willing to defend your choice
Deploys
Everyone Can Deploy
• Automated our deployment process to a single step.
• Everyone can deploy, deployments are logged
• Easy rollback is a requirement!
• Implementing feature flags to turn off single parts of our application.
TestsUnit
Functional Integration
Acceptance etc
Automated Tests
• If you want to trust your Devs, you need tests
• Legacy apps we wrote Integration Tests
• New Apps & Refactored Legacy Parts have Unit Tests
• Continuous Integration to make sure tests run
So, where’s the salt?
Step #4: Devs Use The Ops Tools
Devs can grok salt
Safe Environment For Devs to Learn
Safe Environment For Devs to Learn
salt \* cmd.run "rm -rf /tmp /*"
Safe Environment For Devs to Learn
salt \* cmd.run "rm -rf /tmp /*"
Salt is awesome, but it can’t !recover from that
Dev Salt MasterDevs Can Look Into Every Server
Dev Salt Master• Every server has two minions:
• Admin Salt (aka root)
• Dev Salt (aka bob)
• Each connect to different master server:
• All Devs have access to Dev Salt Master
• Trusted Devs get access to Admin Salt Master
Everything Salty in GitReminder:
Dev Environment
• Developers own the Dev Environment
• Dev Teams manage the Salt States for their Env
• Vagrant + Salt for their Env
• Who makes changes? Developers
• DevOp helps advise & offer support
Team Structure
Dev Ops
Stage Environment• Stage & Production use same salt repos, different
branches
• Developers make all the changes for Application Servers
• All Changes through Pull Requests
• We’ll worry about env changes before code
• Small changes we quickly release, large or long running branches are scary & dangerous
Production Environment
• Merge change to Production Branch
• salt \* state.highstate
• Reminder: Small quick changes over time, never a large change at once.
Environment Caveats• Ops & DevOps Manage VM Hosts, Physical Load
Balancers, FireWalls, etc
• Ops & DevOps manage servers that deal with data:
• MySQL
• MongoDB
• etc
Mentoring Devs
Mentoring Devs
• Not every Dev will become an amazing DevOp
• Thats okay!
Level of “DevOps” Skills
• Thinks about their impact on Ops: Everyone
• Able to debug issues with production: Most
• Able to make changes to environments: Many
• “Awesome DevOp”: Some
Team’s “DevOps” Skills
0
25
50
75
100
Think Debug Change DevOp
Current Goal
So Everything Is Awesome for us, right?
Honesty Slide: We Have Skeletons In Our Closets
Where We Are At
• All Dev Environments using Vagrant + Salt
• All New Stage & Prod Environments are Salty
• Some Legacy Stage & Production Envs are Salty
• Continuously working on getting out stuff salty.
Making This Work For Your Team
Honest Introspection• Determine for your
team what are your… • Strengths • Weaknesses • Problems • Goals
Increase Team’s Insight • Make sure devs can see
& understand how their code performs
• Increase responsibility of team for those metrics.
• If they break it, they fix it. Do not always bail them out.
• Everyone can see everything.
Increase Team’s Insight • Make sure devs can see
& understand how their code performs
• Increase responsibility of team for those metrics.
• If they break it, they fix it. Do not always bail them out.
• Everyone can see everything.
Mentor Those With Desire / Aptitude
• Give Developers Safe Environment to Learn
• Let them submit code-reviewed changes for Stage & Production
• When teaching / mentoring, let the learner drive, kindly offer advice and help.
• It takes time, but worth the investment.
A Few Final Thoughts
Team Culture Matters
Positive Influence
Questions?
Thank YouJustin Carmony
Email: [email protected]: @JustinCarmony
IRC: carmony #salt #uphpuWebsite: [email protected]
p.s. we’re hiring, email / pm / tweet me