New DevOps for the DBA

© 2017 Delphix. All Rights Reserved. Private and Confidential.© 2017 Delphix. All Rights Reserved. Private and Confidential.

Kellyn Pot’Vin-Gorman | Technical Intelligence Manager| October, 2017

The Revolution

DevOps to DataOps

© 2017 Delphix. All Rights Reserved. Private and Confidential.

2

Kellyn Pot’Vin-Gorman

Technical Intelligence Manager, Delphix

• Multi-platform DBA, (Oracle, MSSQL, MySQL,

Sybase, Postgres…..)

• Oracle ACE Director, (Alumni)

• Oak Table Network

• APEX Women in Technology Award, CTA 2014

• STEM education with Raspberry Pi and Python

• President, Rocky Mtn Oracle User Group

• Liaison, Denver SQL Server User Group

• DevOps author, instructor and presenter.

• Author, blogger, (http://dbakevlar.com)


3

• https://en.wikipedia.org/wiki/DevOps

The Source

DevOps is a short for DEVelopment and IT OPerationS. The term is used to

refer to a set of practices that emphasize the collaboration and communication

of both software developers and information technology (IT) professionals while

automating the process of software delivery and infrastructure changes. It aims

at establishing a culture and environment where building, testing, and releasing

software can happen rapidly, frequently, and more reliably

https://en.wikipedia.org/wiki/DevOps


The Life of a DBA

Database management, automating where able

Provide environments and assist with projects

Secure database environments

Optimize data access

Collaborate to solve business challenges


The Life of a DevOps Engineer

Automation through scripting

Orchestration between environments, including data sources

Security

Optimize projects, improving on each pass

Collaborate and Communicate


6

Tools Empower?

• DevOps requires agility

• Agility requires more automation

and tools to ease demands

• Even the history of DevOps

includes tools

• What tools are part of DevOps for

the DBA?


7

How DBAs Risk the Change to DevOps

• Database Administrators heavily influence decisions.

• We’re sometimes identified with manual processing as a way to measure our value.

• Introduction of the cloud, especially and SaaS, (Software as a Service) has been seen as

competitive to our role.

Still, we’re essential to a successful change, or many are working around us…


8

Why Should We Master Tools?

• We Should be part of the development process

from the beginning

• We perform code reviews

• We check in and release code

• We write code, too.

• We create automation scripts.

• We monitor and optimize


Tools of the Development

• Oracle SQL Developer

• DBArtisan

• Redgate SQL Toolbelt and SQL Prompt

• SQL Automation PL/SQL Developer

https://www.idera.com/dbartisan-database-administration-solution

https://www.red-gate.com/products/sql-development/sql-toolbelt/

https://www.red-gate.com/products/sql-development/sql-prompt/


10

DevOps Terms

• Build Automation

• Configuration Management

• Continuous Delivery/Rapid Deploy

• Functional Testing/Unit Testing/Test Driven

Development

• Monitoring/Alerting

• Packages/Containers/Virtualization

• Release Coordination/Orchestration

https://xebialabs.com/glossary/

https://xebialabs.com/glossary/


11

Method to this Madness

• Identify Category/Methodology

• Identify tools

• Some tools will bridge more than one

category or methodology

• Dig into unique or interesting specifics

about the tool


12

Build Automation

• Maven

• Gradle

• Packer

• Cmake

• ANT

DBAs are a natural fit, due to their

history with scripting and automation

experience for backups, monitoring and

alerting.

Build links: Maven, Gradle, Make, Packer, MSBuild

https://maven.apache.org/

https://gradle.org/

https://www.packer.io/

https://cmake.org/

http://ant.apache.org/

mailto:https://maven.apache.org/

mailto:https://gradle.org/

mailto:https://www.gnu.org/software/make/

mailto:https://www.packer.io/

mailto:https://msdn.microsoft.com/en-us/library/dd393574.aspx


13

MS Build

MS Build 15 came out in March 2017 and

part of Visual Studio

• No longer installed in the Global Assembly

cache.

• XML based

• Project file made up of items, properties

and tasks- and of course, targets.


Continuous Delivery

DBAs are much slower to

embrace continuous delivery.

The risk is high, with it’s “fail

first, fail fast” approach. Yet

we are its greatest asset.


15

Jenkins for CI/CD

Jenkins is an open source automation server-

• Easiest executed via a jetty servlet

• Has an easy to access URL, (commonly host:8080)

• Automation is done via individual projects.

• Has numerous plugins to support environments

• Supports multiple scripting languages, including shell, batch commands and Ant scripts.

• Supports robust revision control and compatible with version control systems like CVS and

Subversion.

• Supports notifications like qmail and sendmail.


16

Jenkins Groovy Script Example// prepare variables for subscript execution:

String subscript = "cdmserver-monitor-instance-status.groovy";

println "executing subscript ${subscript}"

// variables to be passed to the subscript

status = (action == "start" ? "started" : "stopped")

def s = ScriptlerConfiguration.getConfiguration().getScriptById(subscript)

File scriptSrc = new File(ScriptlerManagment.getScriptDirectory(),

s.getScriptPath());


17

Jenkins Plugin for Delphix

https://github.com/delphix/delphix-jenkins-plugin

https://github.com/delphix/delphix-jenkins-plugin


18

Configuration Management

Concept & Development Testing & Verification

Operations & Support Deployment to Production


19

Ansible

• Learn about Playbooks

• Powerful collection of steps, (aka plays)

• Plays consist of a list of tasks

• Handlers allow for remote operations

• YAML Syntax

• Security configurations based on hosts and users

• Ability to run as secondary users, (privilege

escalation)

• Huge support community and best practices

http://docs.ansible.com/ansible/playbooks_intro.html

http://docs.ansible.com/ansible/YAMLSyntax.html

http://docs.ansible.com/ansible/playbooks_best_practices.html


20

Example of an Ansible Call

- name: Transfer and execute a script.

hosts: server

remote_user: test_user

sudo: yes

tasks:

- name: Transfer the script

copy: src=test.sh dest=/home/test_user

mode=0777

- name: Execute the script

command: sh /home/test_user/test.sh


21

Puppet

• Review, automate, deploy and invest in all your

software.

• Control and enforce consistency across environments

and platforms.

• Share, test and enforce changes on-premise and in the

cloud.

Similar to Puppet

• Chef

• Jenkins


22

Rapid Deploy

Software Content

Enablement ServiceNeed or

Focus Area

All Tiers of Specific

Environment


23

Release Coordination/Orchestration

A demand for release tools and orchestration was at times a direct result to DBAs hesitation to risk

production with a release. This helped automate and bypass the DBA to get code and features into

production.

• Docker Hub

• Artifactory

• Zephyr

• Octopus Deploy

Deployment Links- RapidDeploy, ElasticBox, JuJu

https://hub.docker.com/

https://www.jfrog.com/artifactory/

https://www.getzephyr.com/test-management/rapid-deployment-tools

https://octopus.com/

mailto:https://rapiddeploy.com/

mailto:https://elasticbox.com/explore

mailto:https://www.ubuntu.com/cloud/juju


24

Octopus Deploy

• Categorized as Continuous Integration Tool

• Release and deploy

• Promote to subsequent environments

• Cross tier/application, etc.


Automation and Orchestration, DB Style

• Idera SQL Admin Toolset

• Redgate DML Automation

• Automic

• CA Automation

https://www.idera.com/productssolutions/sqlserver/sqladmintoolset

https://www.red-gate.com/products/dlm/dlm-automation/


Source Control and Repositories

• Git

• Bitbucket

• GitLab

• Subversion

• Mercurial

https://git-scm.com/

https://bitbucket.org/product?_mid=ec051a8d5bf4616dde12a767df3cb891&aceid=&adposition=1t1&adgroup=9753943967&campaign=177597647&creative=213764064187&device=c&keyword=bitbucket&matchtype=e&network=g&placement=&ds_kids=p19174776695&gclid=EAIaIQobChMIsr7um7-Z1wIVAl6GCh0g5gZFEAAYASAAEgJrxPD_BwE&gclsrc=aw.ds

https://about.gitlab.com/?utm_medium=cpc&utm_source=google&utm_campaign=brand+general+us+desktop+e&utm_term=gitlab_e&utm_source=google&utm_medium=cpc&utm_campaign=&utm_content=&utm_term=gitlab&gclid=EAIaIQobChMIntmisL-Z1wIVSgOGCh0fiAbAEAAYASAAEgLIDPD_BwE

https://subversion.apache.org/

https://www.mercurial-scm.org/


27

GitHub

Git is an open-source version control system that was started by

Linus Trovalds, (the creator of Linux)

• If you aren’t using it now, start. Yes, as a DBA…

• Great option to highlight your scripting work

GitHub Alternatives

• Subversion

• CVS

• Mercurial


28

Creating a GitHub Repository Project


29

Github Repository Example


30

Source/Version Control, but Data?

We all know about version

control with code, but how

are we, as DBAs doing

version control with data?

This is a concept that hasn’t

been discussed much and

DataOps is the new way of

bridging some of this

challenge.


31

The Goals of All Tools are the Same

• Shorter Delivery Cycles

• Higher Success Rate on Releases

• Shorten Time to Address Issues

• Continued Improvement in

Communication

• Increased Revenue


32

Let’s Talk Data Gravity and It’s Impact

• Data is HEAVY.

• Relational databases are heavily weighted and can

be very impacting to DevOps implementations.


33

Oh DBA, to Embrace Virtualization…

A technical approach in which users and applications do

not use physical machines, but simulated systems

running on actual, “real” hardware. Virtualization can be

used to eliminate resource usage and enable savings for

databases, network, file and application management,

along with server infrastructure.


Older Cloning Methods


Virtualize

Compress and Dedup


36

Why Database Virtualization is Different

• Many tools take considerable time to recover.

• DevOps automate an “undo” for development and

testing that includes data rewind.

• Include a self-service tool that will allow for rewind

without DBA intervention.

• Allows for data version control and DataOps, the

next step in DevOps


37

Virtualization Tools

• Delphix

• Veritas Velocity

• Oracle Thin Cloning Features

• Actifio

https://www.delphix.com/solutions/sql-server


▶▶▶ Virtualize and Deployed▶ ▶ ▶

600GB

Virtualization 101 via Delphix

Storage Pool for Delphix

QADEV PATCH TEST

PRODUCTIONDatabase/App Tier

1 TB

1 TB

Read From Production

Each Virtual Database takes up around 5-10Gb upon creation, (dependent upon

parameters)TEST

Read AND Write


Virtualized Database, (VDB)


Data Virtualization: Space Savings

Delphix Virtualization EngineValidated Sync Target

Dozen Virtual Database Copies, (VDBs)


41

The Future is Now- Containers/Data Pods

Containers offer the ability to isolate application code

and/or the whole infrastructure stack into a package able

entity to ease deployment, even from the same kernel.

This is a powerful tool for DevOps to ease deployment for

complex tiers, applications and multiple data stores.

Data Pods is the next step, where a pod is created from

virtualized environments.


42

Docker

• Virtualization is important,

this is OS level

• Resource level isolation at

the kernel level

• Technology such as

OverlayFS, allows for

containers, which removes

overhead of other

virtualization technologies.

Docker Alternatives:

• Open Container

Initiative, (OCI)

• Kubernetes

• CoreOS

• Apache Mesos

https://www.opencontainers.org/

https://kubernetes.io/

https://coreos.com/

https://mesosphere.com/why-mesos/


Group Environments into Data Pods


Why Create Data Pods

• Development is done by project or feature vs. tier or

product.

• With the introduction to the cloud, the business

requires those that can do more at a global scale.

• Ease of management and maintenance

• Provide more value to the business


45

Data Pods

Database

Unstructured

Data

Application


Data Pods for Ease of Delivery

Create “Container”

Rsync (UNIX/Linux)

Robocopy (Windows)

Reporting

NFS

iSCSI

Development

NFS

iSCSI

Testing

NFS

iSCSI

Delphix Virtualization EngineSame Storage Size as Production

Deployed using Jenkins


47

How Does This Improve Agile?

Finance

HR

CRM


48

Timeline in a Project

Waterfall- 5 Releases

January February March April

Agile with Virtualization- 12 Releases

January February March April

Database Refresh

Development Production Release

Testing Cycle

The Goal of

Agile

Data Holdup to Development

Cycle


Data Versioning

• Delphix Jet Stream

• Redgate SQL Data Compare

• Oracle SQL Developer

https://docs.delphix.com/docs/delphix-jet-stream

https://www.red-gate.com/products/sql-development/sql-data-compare/


50

Branching and Bookmarking

• The ability to mark each iteration of development

with a bookmark

• Simplify to lock and deliver while testing a

consistent image via a virtual database, (VDB)

• If a something goes wrong, the ability to

“bookmark”, (and subsequent snapshot) to

deliver to development to address.


51

DevOps and Source Control


Administration Ease


Self-Service

Release 1.2 Release 1.3 Branch 1.3

Branch 1.1


Methodologies and Collaboration


55

Scrum

https://www.scrumalliance.org/community/articles/2014/april/devops-and-agile

https://www.scrumalliance.org/community/articles/2014/april/devops-and-agile


56

Lean/Kanban


57

Extreme Programming


58

Crystal

Alistair Cockburn said, “Crystal is a family of human-powered, adaptive,

ultra light, ‘stretch-to-fit’ software development methodologies.”

• People are the most important aspect of Crystal.

• Development is self-sufficient and self-organizing, (very different from

Scrum)

• Very lightweight

• Very little documentation, overhead management or reporting.


59

Feature Driven Development


60

Collaboration Practices can Mutate

• Methods- Scrum, KanBan, Extreme Programming, Continuous

Delivery

• Official hybrid approaches like Scrumban and KanPlan

• Numerous other combinations created from any of the

originals.

mailto:https://www.scrumalliance.org/

mailto:http://atlassian.com/agile/kanban?_ga=2.92098330.1868173152.1496357560-1545087632.1496357560

mailto:http://www.extremeprogramming.org/

mailto:https://continuousdelivery.com/


• All IaaS solutions provide encryption in-flight and encryption at-rest• But encryption doesn’t protect data as much as it needs to be .

• Europe already requires data masking, not just data encryption for any confindential data, (GDPR):

• http://ec.europa.eu/justice/data-protection/article-29/documentation/opinion-recommendation/files/2014/wp216_en.pdf

Confidential data

http://ec.europa.eu/justice/data-protection/article-29/documentation/opinion-recommendation/files/2014/wp216_en.pdf


• Encryption is reversible data obfuscation, which is very different from masking data.

• Data masking is non-reversible.

• It solves the issue at the data level.

• Is authentication and authorization in non-production in compliance with security goals?

• All organizations will soon need to review if critical data in non-production environments be accessible to developers, testers and users.

Confidential data


Encryption and Masking

• Optim

• Delphix Masking

• Dataveil

• CA Test Data Manager

• Oracle Encryption and Masking

https://www.delphix.com/platform/masking

http://www.dataveil.com/

https://docops.ca.com/ca-test-data-manager/4-2/en/provisioning-test-data/mask-production-data/data-scrambling/generate-masking-scripts-for-sql-server


• Masking personally-identifiable, (PII, HIPPA, PCI, etc.) information renders it useless from a security standpoint

• Resolves both the technical and personal responsibility issue.

• The data can be masked before it moves to non-production, removing unnecessary risk.

Why Masking is Part of the Answer


Confidential data

Production

Non-Production

Exposure


Confidential data

Production

Non-Production

Exposure

Encryption

Masking

Solution


Auditing Solutions

• Idera SQL Secure and SQL Compliance

Manager

• Redgate SQL Source Control

• Oracle Auditing Features

https://www.idera.com/productssolutions/sqlserver/sqlsecure

https://www.red-gate.com/products/sql-development/sql-source-control/


Data Causes Friction

• If those managing the data sources aren’t included and working well with those needing that data to produce features and products, friction is the result.

• DBAs are consistently working with archaic utilities that guarantee outcome, rarely speed of results.

A DBA is only as good as their last backup.


What is Data Gravity?

Data Gravity is the ability for data to attract applications, services, etc. As with the laws

of physical gravity, data, due to its mass, will attract all else that has less mass.


When you consider…

An estimate of a 10% increase in data accessibility will result in more than $65 million in additional net income, you can see why data

gravity is a big deal.


How much data are we talking about?

• More data has been created in just the last two years than the previous history of humanity

• 1.7MB of new information created every second per human on the planet.

• Introduction of big data often has same development pain points.


74

Know Your Tools

Incredible variation of tools and knowing tool classifications. Use

the right tool for the job and recognize that each tools has its

own “flavor.”


Remove the Roadblocks

• Remove the data as the roadblock.• Use virtualization and containers to simplify.• Self-service portals for developers and testers to

refresh and develop they way they do in an agile environment.

• Tools with an interface made for Agile and Development data at its focus.


76

Review- DBA Focus Areas

• Update skills, (scripting, monitoring, automating) to remove bottlenecks

• Enhance monitoring utilities that take infrastructure and development tools into

overall view

• Recognize a need to become more skilled in scripting and become

knowledgeable of DevOps tools.

• Data is EVERYWHERE! Recognize the integral part Database Administrators

play in DevOps and learn more about the next generation- DataOps!

Don’t fear the future, embrace it.

DBA

DevOpsSkills


77

The Periodic Table of DevOps Tools

https://xebialabs.com/periodic-table-of-devops-tools/

https://xebialabs.com/periodic-table-of-devops-tools/


78

Read- The Phoenix Project and the DevOps Handbook!

Delphix with SQL Server- the Basics

https://docs.delphix.com/docs/delphix-administration/sql-server-environments-and-data-

sources/managing-sql-server-environments/overview-of-setting-up-sql-server-

environments

Delphix Upgrade Workflow: https://community.delphix.com/delphix/topics/tip-of-the-day-

upgrading-a-sql-server-dsource

Upgrading the Dsource after an Upgrade: https://docs.delphix.com/docs/delphix-

administration/sql-server-environments-and-data-sources/virtualizing-databases-using-

delphix-with-sql-server/managing-sql-server-dsources/additional-dsource-topics/upgrading-

a-dsource-after-a-sql-server-upgrade

Delphix in the Cloud

https://www.delphix.com/solutions/cloud-migration-virtual-data

References and Tips

https://www.amazon.com/Phoenix-Project-DevOps-Helping-Business-ebook/dp/B00AZRBLHO/ref=sr_1_1?ie=UTF8&qid=1502559091&sr=8-1&keywords=the+phoenix+project

https://www.amazon.com/DevOps-Handbook-World-Class-Reliability-Organizations-ebook/dp/B01M9ASFQ3/ref=sr_1_1?s=digital-text&ie=UTF8&qid=1502559124&sr=1-1&keywords=The+DevOps+Handbook

https://docs.delphix.com/docs/delphix-administration/sql-server-environments-and-data-sources/managing-sql-server-environments/overview-of-setting-up-sql-server-environments

https://community.delphix.com/delphix/topics/tip-of-the-day-upgrading-a-sql-server-dsource

https://docs.delphix.com/docs/delphix-administration/sql-server-environments-and-data-sources/virtualizing-databases-using-delphix-with-sql-server/managing-sql-server-dsources/additional-dsource-topics/upgrading-a-dsource-after-a-sql-server-upgrade

https://www.delphix.com/solutions/cloud-migration-virtual-data

© 2017 Delphix. All Rights Reserved. Private and Confidential.© 2017 Delphix. All Rights Reserved. Private and Confidential.

Kellyn Pot’Vin-Gorman

Technical Intelligence [email protected]://dbakevlar.com

mailto:[email protected]

http://dbakevlar.com

New DevOps for the DBA

Technology

Transcript of New DevOps for the DBA