How Database DevOps Will Help You

To Stay Ahead Of The Competition

Derek Campbell, Solutions Architect at Octopus Deploy

Agenda

• Background

• DevOps

• Benefits

• Roadblocks

• Questions

Background

Previous Work Experience

• Release Manager

• Lead DevOp

• System Architect

• System Administrator

What are the issues we can solve?

• Databases out of pace with application development

• Lack of Traceability

• Inability to Roll back

• Databases are the bottleneck

• Releases less frequent with increased risk

• Efficiency – Manual work takes more time.

System Development Life Cycle

Development Team

Operations Team

Knowledge Silos

• Development Team• Development roadmap

• Feature requirements

• Application specific knowledge

• Operations Team• Performance statistics

• Support tickets submitted

• Infrastructure specific knowledge

Typical Communication Between Teams

• “This service is down please fix it”

• “Your service is consuming 16 GB of RAM, fix it now”

• “I have a call from a customer and they say your application is

slow”

• “I got called out last night because this SQL Job failed. Fix it.”

• “It worked in Dev”

A Lack of Trust

Lack Of Trust

Slows Everything

Down

It doesn’t need to

be Perfect!!

Example

• Application ready for deployment

• Development: • “We agreed with the customer to deploy a new feature next week”

• Operations:• “Every time you deploy your application you increase our support

tickets volume”• “Every time you deploy your application consumes more resources”

• New Policy: Min 2 Week Notice For Deployments

Massive Deployments

• Multiple Sprints prior to going to production

• Multiple Features bundled together prior to going

to production

• Long Waterfall release cycles (quarter or months)

Perfect Storm

• Quarterly Releases

• Lots of changes

• Out of sync databases

• Manual Deployments

Everyone Is Unhappy

• Business Owners• “This isn’t going to work now, we need to add the following

features to make this viable”

• “Fine. Fine. Fine. Deploy it as is. I’ll add it my next release doc”

Everyone Is Unhappy

• Users• Not what they wanted.

• How long until the application is no longer relevant

• Head to competitors

Everyone Is Unhappy

• Security• Data Compliance

• Patches

DevOps

DevOps: Development and Operations

Teams Working Together Towards a Common

Goal

Dev + Ops = DevOps

Work toward a Common Goal

• 1 deployment a month

• 1 deployment a week

• Zero Downtime Deployments

• 10 deployments a day

• Fixing all critical bugs within 2 business days

Common Goal

• Performance tests before each release

• Decrease deployment time

• Increase response time on critical methods

• Decrease resource usage

Example: Performance Monitoring

• Dev Team Problem: Bad Performance

• Ops Team Problem: Babysitting Apps

• DBAs: Increase in resource consumption

• Common Problem: What is causing the slow

down

Example: Performance Monitoring

• Solution: New Relic• Ops: Train developers on how to use NewRelic

• Developers: Train Ops and DBAs what to look for (sprocs, stack traces, etc)

• DBAs: Look for slow running sprocs, or sprocs getting hit thousands or millions of times

Example: Automated Database Deployments

• Dev Team Problem: Hours spent manually creating delta scripts

• DBA Problem: No consistency between teams or environment databases, no auditing of scripts, no testing of scripts.

• Common Problem: 30 minutes - 2 hours per deployment per team

• Solution: Use Redgate tooling with Octopus Deploy to automate database deployments

• Result: • 5 - 20 minute deployments• One release a quarter down to one release every 10 days

Benefits

Speed - Release When Ready

• At Octopus Deploy we release as soon as the feature is ready

• Monthly releases of major features, daily releases of bug fixes, performance improvements, and UI improvements

• Use “release bots”, TeamCity, Slack and Octopus Deploy to deploy Octopus Deploy

Example - Fixing Critical Bug

• Customer reported bug to their process through

support channels at Octopus Deploy Tuesday

morning

• Discussed bug during the day

• Fix released overnight

• In customer’s hands by Wednesday morning

Example - Better Performance

• Application averaging 2000 requests per minute

• After deployment average response time went

from 20 ms to 200 ms

• Identify slow performance and deploy release

before first support ticket came in

Staying Ahead Of Competition

• Respond to customer feedback

• Fix critical bugs

• Experiment and A/B testing

• Keep company relevant

• Faster time to market

Staff

• Higher staff engagement

• More interesting work

• Better work/life balance

• Cross Skilling and Self Improvement

• Decentralized responsibility

Compliance

• Auditing

• Standardization

• Reduce Errors

• Protect Sensitive data

• Increased Trust

Getting Started

Start Small

• Attend meetings

• Discuss issues and complaints

• Listen and Learn

• Pick a common problem

• Use existing tooling

Go a bit bigger

• Solve a larger problem

• Pilot Application / Group

• New Tooling

• New Processes

Tooling

• Build: TeamCity / VSTS / Jenkins / Bamboo

• Unit Testing: NUnit / MSTest / XUnit

• Integration Testing: Selenium / Postman

• Deploy: Octopus Deploy / Redgate (for DBs) / VSTS Release / DACPAC

• Monitoring: NewRelic / DataDog / DynaTrace / SQL Sentry

• Error Logging: Splunk / Seq / PaperTrail

DevOps is People,

Processes and

Tools

Roadblocks

People

• “We’ve been doing it this way for 10 years”

• “Sure what we have is a little bumpy, but it works”

• “I’m too busy to change”

• “This was tried once before and it failed”

• “What you proposed won’t work for me or my

team, we’re special”

Time

• Time to implement changes

• Each new change requires multiple iterations

• Deadlines and other conflicting priorities

Cost

• Tools cost money

• Cost per hour

• Consultants

• Training

• Return of Investment

Breaking Through Roadblocks

• Technology is the easy part

• Pilot team / project

• Small changes

• Don’t aim for perfection at first

• Provide clear and measurable stats

• Be Transparent

Anyquestions?

Thank You!