Database Lifecycle ManagementAchieving Database Continuous Delivery

Grant Fritchey and Matthew Skelton

This book is for sale at http://leanpub.com/database-lifecycle-management

This version was published on 2016-03-11

ISBN 978-1-910035-05-4

This is a Leanpub book. Leanpub empowers authors and publishers with the Lean Publishingprocess. Lean Publishing is the act of publishing an in-progress ebook using lightweight tools andmany iterations to get reader feedback, pivot until you have the right book and build traction onceyou do.

© 2015 - 2016 Redgate Software

Tweet This Book!Please help Grant Fritchey and Matthew Skelton by spreading the word about this book on Twitter!

The suggested tweet for this book is:

Checking out ”Database Lifecycle Management: Continuous Delivery for Databases” by@GFritchey & @matthewpskelton

The suggested hashtag for this book is #DLMBook.

Find out what other people are saying about the book by clicking on this link to search for thishashtag on Twitter:

https://twitter.com/search?q=#DLMBook

Contents

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iA Word on LeanPub . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i

In a world without effective Database Lifecycle Management . . . . . . . . . . . . . . . . iiWhy you need this book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiT-R-I-M database processes to reduce risk . . . . . . . . . . . . . . . . . . . . . . . . . . ivThe book sections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v

Section 1 - DLM foundations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viSection 2 – Enabling DLM in the Enterprise . . . . . . . . . . . . . . . . . . . . . . viSection 3 - DLM for the relational database . . . . . . . . . . . . . . . . . . . . . . viiSection 4 – DLM beyond Development . . . . . . . . . . . . . . . . . . . . . . . . . viiSection 5 - DLM and the data ecosystem . . . . . . . . . . . . . . . . . . . . . . . . vii

Who should read this book? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiResources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix

Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi

DLM Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1. Databases and Change . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2. How DLM works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

3. Core DLM Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Enabling DLM at the Enterprise Level . . . . . . . . . . . . . 5

4. How CIOs/CTOs can enable DLM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

5. Team structure/challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

6. Core practices to enable non-breaking changes . . . . . . . . . . . . . . . . . . . . . . 8

7. Operational Considerations: the “operations first” approach . . . . . . . . . . . . . . 9

CONTENTS

DLM for relational databases . . . . . . . . . . . . . . . . . . . . . . 10

8. Version control for DLM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

9. Build . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

10. CI for Databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

11. Testing Databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

12. Release / Deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

13. Monitoring and Feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

14. Issues tracking for DLM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

DLM in the Enterprise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

15. Configuration Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

16. Access Control / Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

17. Audit, Traceability, Compliance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

18. Documenting Databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

19. Reporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

The Data Ecosystem, and Extending DLM . . . . . . . . . 24

20. Hybrid database environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

21. Operational Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

22. DLM for ETL Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

23. Content Management Systems (CMS) . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

24. Business Intelligence (BI) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

25. Data Retention, Discovery and Curation . . . . . . . . . . . . . . . . . . . . . . . . . . 30

Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

IntroductionA Word on LeanPub

As we are developing this book in an iterative, ‘Agile’ way, the content is absolutely going to changeand evolve as we progress. As a result, the outline you see here today is still subject to change,evolution and improvement. We think it’s a pretty good reflection of what would be useful to knowwhen you’re trying to improve your database lifecycle management, or get buy in for the idea, butwe’d love to here your thoughts on what we’re proposing.

At the end of each chapter, you’ll a call for feedback like this with links to our leanpubfeedback page¹ and also an email address directly to the authors and editor - DLM-Book@red-gate.com². We’d love to hear from you. If you spot errors, issues, things youthink we’ve missed, or just a point you think is worth investigating, please do let us know.

¹https://leanpub.com/database-lifecycle-management/feedback²mailto:DLMBook@red-gate.com

i

In a world without effective DatabaseLifecycle Management

It is 10am in the offices of InfiniWidgets, Inc., a company that originally producedand shipped widgets for use in the food industry, and that now offers a SaaS-basedmanufacturing plant tracking system for its clients. The company is successful andhas a large database with orders and inventory records dating back to its inception,17 years ago. Its four software development teams work on various different aspectsof the internal Order and Inventory (OI) system, as well as customer-facing FactoryManagement (FM) system.

Jess, the lead software developer from the FM team, and acting Scrum Master, isfrustrated. The database changes were supposed to go live today but the scripts arestill under review by the database administration (DBA) team. She’s had no update viaemail or the ticketing system to tell her when the changes will be ready. She informsthe project manager of the delay, who fires off an impatient email to the DBA team.

Bal, in the DBA team, is multi-tasking as fast as he can. He’s just finished some urgentschema updates that the OI team requested yesterday, and it took longer than he hoped,as the scripts were a bit of a mess. He’s now working on a ticket from the FM team andthey are not happy about the delay. His heart sinks when he sees how the numerousrequested index modifications and additions are likely to affect the performance oftransactions on the FM system.

He fires off a mail to Jess explaining then problem and attaching the performance chart,wondering aloud why he never gets to see these proposed changes until the last minute.If he’d seen them earlier he could have warned them that this approach wouldn’t work

Jess views Bal’s email and sighs in exasperation. “Why couldn’t we have had thesemetrics before? We might have taken a different approach!” When Jess explains toher project manager that the DBA team have rejected the change, he loses his temper.“How many times do we have to put up with these DBA roadblocks?” He fires off anangry email to the Head of Delivery before marching up to Sam, the lead DBA, andinsisting that the FM team’s changes be deployed that day in order to meet their clientcommitment.

Under pressure from all sides, Sam feels there is no choice but to deploy the databasechanges from Jess’s team, even though Bal tells him they are likely to cause problems.Sam instructs Bal to run the scripts after 6pm that day.

The next day, the problems begin. The DBA team begins to receive emails from thecustomer support team saying that five of their main six customers are having problems

ii

In a world without effective Database Lifecycle Management iii

accessing their factory management portals. The problem escalates; queries are hangingdue to severe blocking, users are seeing queries fail due to deadlocks. Eventually thewhole site goes down.

A frustrating half-day and hundreds of emails later, the DBA team breaks the bad newsthat they need to restore the database from a nightly backup and the customer supportteam spend the next three days re-entering lost customer transactions.

If this tale, fictional as it is, strikes notes of familiarity with you, then you should find all or parts ofthis book helpful.

Why you need this book

If you work for an organization that produces database-driven software system, and the databaseneeds frequent or periodic changes to features, functionality, or data, then this book is for you. Itdescribes how to apply the business and technical approaches of Database Lifecycle Management(DLM) to make all database processes more visible, predictable andmeasurable, with the objectiveof reducing costs and increasing quality.

Modern organizations collect large volumes of data, in a range of formats and from a range ofsources. They require that data to be available in a form that allows the business to make fastdecisions and achieve its strategic objectives. The DLM approach acknowledges the dual reality thatdatabases are becoming increasingly complex and challenging to develop and maintain, in responseto the business’s demand for data, and that we must be able to adapt even complex databases inresponse to changing requirements.

Given these demands, traditional manual approaches to database changes become unsustainable andrisky. Instead, we need to increase automation whilst retaining and using the expertise of peoplewho understand our organization’s data. We need team structures that encourage communication.We need a framework within which to make simple, stepwise improvements to our core databasebuild, test and delivery processes, and then to apply the workflow that stitches together theseotherwise disparate processes into a coherent, automated pipeline to database delivery. We needto make databases easier to maintain, using instrumentation, and by considering their supportrequirements as part of the design process.

DLM will help you achieve these goals. The scope of DLM is, as the name suggests, the entire usefullife of a database. DLM starts with aspects of design and data architecture, encompasses the databasedevelopment and delivery processes, and extends to the support and maintenance of the databasewhile it is in operation. It aims to ensure that an organization has in place the team, processes,methods, business systems and tools that will allow it to design, develop and then progressivelyrefine even the most complex databases, with the minimum difficulty.

This book will explain how to use DLM to:

In a world without effective Database Lifecycle Management iv

• Automate many of the critical processes involved in the design, development, delivery andongoing operation of a database.

• Improve the organization’s knowledge of the database and related applications• Identify opportunities for optimization• Encourage technical and organizational innovation.• Ensure that the database supports your key business goals, and drives strategic decisionmaking, within the enterprise.

The overall goal is to help you evolve databases by a process of continuous, incremental change,in direct response to the changing data requirements of the business, and to improve the delivery,operation and overall quality of your database systems.

T-R-I-M database processes to reduce risk

Making changes to database-coupled systems can often be difficult enough within organizations, butwhen those changes need to be made more frequently, more rapidly, and more reliably traditionalmanual approaches to database changes become unsustainable and risky. Historically, organizationswho rely on these manual approaches have delayed or avoided making changes to database-coupledsystems due to the perceived risk and complexity.

However, organizations that lag behind in the technical practices that allow change will find them-selves at a disadvantage. Their competitors innovate quicker. Funding is moved to organizationsthat can make more effective use of scarce resources. The speed of change in the technology sectorleaves them at the mercy of legacy, unsupported technologies, costly outdated billing models, andeven security vulnerabilities.

Throughout the book, we describe a range of critical database processes in the database lifecycle,from initial design, governance, development, testing, release, deployment and through to ongo-ing operations and maintenance. We identify typical problems associated with ad-hoc, manualapproaches to each process, alongwith the associated business risks.We highlight common technicalissues, bad practices and behaviors and risks, depending on how the team tackles that process, andthen describe how stepwise application of DLMmethods help tame and then systematically improvethat process, making it more Traceable, Repeatable, Improvable and Measurable (T-R-I-M):

• Traceable/Visible – processes are visible to all teams, and to the broader business, from veryearly in the project.

• Repeatable – processes are automated, tested and therefore predictable and repeatable• Incrementally Improved – processes connected by workflows that provide the necessaryinformation to drive continuous short cycles of improvement

• Measurable – processes are instrumented and logged so that errors and deviation frompredictable behavior are corrected quickly

In a world without effective Database Lifecycle Management v

We start from ad-hoc, manual approaches to a particular database process (such as testing) andmovelogically towards an approach that is automated, integrated, measured and standards-compliant.This progression gradually reduces risk, by making the process more visible, predictable andmeasurable. The approach is based loosely on the idea of a “Capability Maturity Model” (describedin more detail in Chapter 1).

As a by-product of this approach, we arrive at a structured series of problems and associatedsolutions. For instance, in the chapter on “DLM for ETL systems” we see:

• Challenge:Constant changes to the ETL data format, by the data supplier, hampers the team’sability to improve the quality of the ETL processes

This challenge has several solutions, including:

• Solution: Insist on data format/schema standards for ETL

We start from ad-hoc, manual approaches to a particular database process (such as testing) andmovelogically towards an approach that is automated, integrated, measured and standards-compliant.This progression gradually reduces risk, by making the process more visible, predictable andmeasurable. The approach is based loosely on the idea of a “Capability Maturity Model” (describedin more detail in Chapter 1).

In this way, we hope to make it easier to:

• For a given skill or method, identify where you are now, the associated risks, and what to aimfor next in terms of process improvement

• Gain rapid access to solutions to specific problems, without having to cover the precedingmaterial in depth

For a given skill or method, it means you can identify where you are now in terms of yourcurrent approach, and the associated risks, and then see what to aim for next in terms of processimprovement. It’s a structure that we hope also provides fast access to solutions to specific problems,without having to cover the preceding material in depth. You may even find it beneficial to printout the Contents section of a chapter on large-format paper in their team area to help with trackingprogress with DLM.

The book sections

The book splits down broadly into five sections.

In a world without effective Database Lifecycle Management vi

Section 1 - DLM foundations

Section 1 describes exactly what we mean by Database Lifecycle Management. We start bydiscussing the key technical, organizational and philosophical ‘barriers’ to making the database,relational or otherwise, a willing participant in practices such as continuous integration, test-drivendevelopment, and minimum viable product, which enable incremental change, and the continuousdelivery of new functionality to the users

Next, we define precisely what we mean by Database Lifecycle Management, and its benefits. Wedescribe the key DLM skills, or working methods, which will support a continuous and incrementalapproach to database change, establish predictable delivery times and result in higher quality, moremaintainable database applications. We also describe, broadly, the data and system architecture,design approaches, team structures and tools that can help with adoption of DLM techniques.

We finish the section with a real life DLM case study, from Frazier Kendrick, called Stumblingtowards Database Change Management. Frazier worked for 15 years as a DBA and IT Managerin the insurance and financial markets. He describes a world in which database change processeswere chaotic at best, where IT improvements were funded only when there was little alternative,rather than as part of a longer-term strategic plan, and teams succeeded through periods of ‘heroicendeavor’, in response to very short-term priorities and targets. Nevertheless, despite this, whendid witness a slow journey towards more managed and optimized systems, where database orapplication changes could be made relatively quickly, and with less disruption, lower risk and fewersurprises, and a semblance of proper IT governance was a reality, with the development team moregenerally accountable to the business as a whole, for the changes they wished to make.

Section 2 – Enabling DLM in the Enterprise

Section 2 explains what sort of thinking and structures that must be present at the organizationallevel in order to enable effective DLM. Like any form a significant organizational change DLMcannot be purely a grassroots, bottom-up process. You’ll need support from whoever is the technicaldecision maker, you’ll need to consider the team structures that will support DLM most effectiely,the personalities within your team, and how your teams are rewarded and for what.

DLMmeans arriving at a set of practices that help us to manage the entire lifecycle of our databases,not just the development cycle. We need to think about maintenance and operational support, andthe needs of other teams within the organization, such as those responsible for data analysis. Weneed to consider what an “operations first” approach really means to the design and developmentof our database applications. It means we need to look beyond the immediate world of SQL scriptsand database backups to the approaches required to support data analysis/mining systems, to issuesof data strategy and traceability for audit, and much more.

We close section 2 by reviewing, at a high level, some of the core techniques, such as use of versioncontrol and well-defined interfaces that decouple application and database, without which any DLMeffort will flounder .

In a world without effective Database Lifecycle Management vii

Section 3 - DLM for the relational database

Section 3 explains in deeper, strategic detail how to apply DLM techniques to the design,development, release and deployment of the relational database, which for many decades, has theheart of an organization’s data.

It will describe the database design, build, test and deployment processes that we need to consider,in order to manage to the database schema, data, and metadata for a database, along with the DLMtechniques and associated workflow that will allow the team to tackle each process in an automated,integrated and measurable way.

Section 4 – DLM beyond Development

Sections 4 looks beyond the foundational DLM practices that are essential to building a databasedevelopment and delivery pipeline, to the broader needs of the organization. It will considerthe fundamental requirements of access control and configuration management that are essentialwhen deploying to production. It will discuss the need for sustainable automated testing andfor traceability, reporting, documentation and so on. In particular, it will consider the topics ofgovernance and compliance, and to establish accountability and visibility for any changes that affectthe organization’s data.

Section 5 - DLM and the data ecosystem

Section 5 considers the applicability of the practices and methods described in Section 2 to thebroader “data ecosystem”, to the ETL process that supply the data, BI systems that must analyzeit, as well as to the design, development, delivery and maintenance of non-relational databases andother supporting, non-transactional systems.

We describe relevant DLM techniques within organizations that have experienced a proliferation ofdifferent types of databases, such as graph-, column-, and document-oriented databases, alongsidestructured, relational data. This is often as a result of changes in development practices, such asadoption of application architectures based around containers and microservices, as well as changesin the type and volume of data that must be collected and analyzed.

Section 5 closes by asking the question “where next?” What new data challenges are on the horizon?What are the emerging new technology will enable our data systems to cope with them?

Who should read this book?

We have tried to write a book that appeals to anyone with an interest in making databases changessmoother, faster, and more reliable. As far as possible, we avoid industry jargon in favor of simple,straightforward terms and descriptions.

In a world without effective Database Lifecycle Management viii

This book offers practical but strategic-level advice on how to tackle the various aspects of databasechange automation, as illustrated by the Problem-Solution approach described previously. It doesnot contain hands-on examples, extensive code listings or technology-specific walkthroughs. Wewill not describe, for example, how to go about setting up a CI Server using a specific tool suchas TeamCity, although we will present the various alternatives, pros and cons where possible andrelevant, and resources for further details.

Database Lifecycle Management encompasses a broad set of practices, requires collaboration andcoordination of activities across many different parts of the organization and therefore involves abroad cross-section of an organization’s IT staff. As a result, this is a slightly uncommon book inthat it is aimed at several different readers, and should probably be read slightly different dependingon who you are and what you need.

When writing this book we have kept in mind three broad areas of responsibility, and typicalassociated job roles:

Responsibility Typical roles

Governance CTOHead of Programme ManagementHead of IT

Operations/Administration Database AdministratorsRelease ManagersSystems Administrators

In a world without effective Database Lifecycle Management ix

Responsibility Typical roles

Software Development Database and ApplicationDevelopersTestersProject Managers

Throughout the text we use these icons to indicate content of particular relevance to each broadcategory of job group. However, we expect that people from any of these roles will benefit fromreading the entire book because each group understanding the needs of other groups will bebeneficial for all.

We have used the established technique of developinguser personas in order to validate thematerialand focus of the different parts of the book. Appendix A provides a series of full-page Personas,describing our understanding of the knowledge, motivation and goals for each of the main job roles.You can also download these personas in PDF format from: http://TheDLMBook.com/resources/(under development).

You may find that the personas help to distinguish better the needs of different groups within yourorganization, and so adapt your working practices to better meet their needs.

Resources

As the book progresses, we will make available various downloadable resources from the Resourcespage of the (eventual) book website: http://TheDLMBook.com/resources/. Feel free to downloadthem, and adapt them as relevant for your organization.

Resource Description URL Last Updated

DLM Capability Matrix A skills-versus-maturity matrix for DLM. Eachcolumn identifies a required skill, and each of 4 rows aprogressive level of maturity, describing the goals,practices and behaviors that characterize each level

Coming Soon

Personas Profiles of several key job roles in terms of theirknowledge, motivations, goals with regard to DLM

Coming Soon

In a world without effective Database Lifecycle Management x

Also on the Resources page, you’ll find an expanding list of useful “Further Reading” references,including several that were an inspiration in the writing of this book, including:

• Continuous Delivery by Jez Humble and David Farley• Release It! By Michael Nygard• Scalable Internet Architectures by Theo Schlossnagle• Agile Database Techniques by W. Scott Ambler• Refactoring Databases by W. Scott Ambler and Pramod Sadalage• Fearless Change and More Fearless Change by Linda Rising and Mary Lynn Manns.

Terminology• BI - Business Intelligence• CI - Continuous Integration - see ???• CD - Continuous Delivery• CMS - Content Management System• DBA - database administrator• Dev - software developer• DevOps - a set of practices for building and operating business software systems thatemphasises strong cooperation between Dev and Ops

• ETL - Extract-Transform-Load, a data processing paradigm for processing data from one formto another

• ITSM - IT Service Management• OLAP -• OLTP - Online Transaction Processing, a kind of database processing found in transactional,interactive systems

• Ops - IT Operations• snowflake - a piece of technology (such as a server or database) that is unique and cannot bereproduced (akin to a snowflak), usually because it has been built manually

• TDD - Test-Driven Development, a pattern for evolving code where we write the test for thecode before the code itself. This acts as a safety harness for changes and keeps code testable.

• VCS - version control system (sometimes called source code control or source control)• VM - virtual machine

xi

DLM Definitions

Before we can make any progress at all, we need to define some terms, and convince you to comealong on this journey with us. We’ll start by explaining what Database Lifecycle Management(DLM) formally is, why we think it’s important, and what it looks like in practice.

1

1. Databases and ChangeAs a start, we’ll frame the major (and all too common) challenges that occur when companiestry to make changes to their relational databases (let along make those changes at speed!), and howwe think the world could look better.

• Common Database Development and Deployment Issues• Maintaining state• Common database deployability blockers and issues¹• Key technical pain points• How does the organization suffer? Can’t adapt• Change is the new black – we need to adapt our adapt processes to cope with regular change

¹https://www.simple-talk.com/sql/database-administration/common-database-deployment-blockers-and-continuous-delivery-headaches/

2

2. How DLM worksSo we’ve established that relational databases are challenging when it comes to publishing changesand maintaining state, but we’ve not yet actually seen what the alternative process might be. Beforewe can proceed, we need to make sure we’ll all on the same page.

• What is the Database Lifecycle?• What DLM is and how it improves the way we thing about / design / work with databases.• How are WE defining its benefits (introduce TRIM)

3

3. Core DLM PracticesThere are a few essential principles that make up the heart of DLM. Concepts that, if they wereremoved from the framework, would leave it significantly poorer, and perhaps even ineffectual. Asyou’ll see, they all centre around the dual goal of delivering value to customers faster, while alsoreducing the total amount of risk in a given project:

• Small changes• Automation• Testing - TDD• Instrumentation• Workflow tools/techniques• Infrastructure automation aka infrastructure as code

– See Matt Wrock - provisioning from bare metal / VM for SQL Server install– Glenn Berry could advise on this chapter (or write it?)

• Small, fast feedback cycles.

4

Enabling DLM at the Enterprise Level

Now that we’ve talked about what DLM actually is, and how it can dramatically improve theway you work with your databases, the next step is to explore how we can prepare ourselves tosuccessfully start implementing it in our organisations.

5

4. How CIOs/CTOs can enable DLMLike any form a significant organizational change DLM cannot be purely a grassroots, bottom-upprocess. You’ll need to get some buy-in and support from whoever is the technical decision makerin your organization, if only because this kind of change takes time, and it can initially be quitedisruptive. If you are a CIO or CTO, or you’re about to talk to one, there are several where areaswhere a hugely positive impact can be made.

• How to get support from your technical decision maker• The broader architectural questions – choosing technology to enable change• Planning to enable change• Ensuring good Governance• The Enterprise Data Model – an organization-wide understanding of the data model (seeRichard M’s case study for the pain that arises when this doesn’t exist)

6

5. Team structure/challengesOne thing that is tacitly understood about many of these broad changes to technical processes, butnot always explicitly addressed or explored, is the fact that they are at least as much about thehuman elements of your organization as the technical ones. By that we mean your organizational orteam culture and structure, the personalities within your team, how your teams are rewarded andwhat for. Essentially, DLM is about technology, but that technology is driven by people, so we needto understand how our teams are going to impact our processes and be aware of how best to adjustthose teams to get the most benefit from any technical change.

• Overcoming different “mental models” for the database across different teams• How different personas think differently about and interact with the database,• Issues this can cause• Team structures that enable DLM• Team-related things that make it easier to change things (in general) in the future• Understanding the effect of Conway’s Law• Silos and responsibilities• Motivations – especially financial things like bonuses• Patterns such as DevOps• Avoiding ‘handover’ and ‘signoff’

7

6. Core practices to enablenon-breaking changes

An essential skill that you will either learn or exercise as part of your DLM journey is how to deploychanges to your production databases in a way that doesn’t break either the application or yourcustomers’ experiences. Some of these we’ll go into more detail on shortly, but the concept is crucialenough that we think it deserves some attention before we go any further.

• Versioning – trunk—based• Versioning your database with your application• Versioned interfaces – Coupling/decoupling of application and database builds• The good and bad of ORM• When to synchronize changes• When to separate changes• BDUF versus Agile for Database Design – presumably we at least start with a rational designbased on normalization standards etc

• Database Refactoring: Defensive Database Design, strategies for non-breaking changes.

8

7. Operational Considerations: the“operations first” approach

Chris and I saw this as an introductory chapter explaining the problems you’ll run into if you don’tconsider any operational considerations – maybe followed later by a more in-depth chapter onhow TRIM can ensure the dev/deign team have the operational details and feedback they need.Part of this is releasing small changes early and frequently outside the dev environment - DatabaseSecurity/configuration management - This is essentially what the “DCMA” in Figure 1 is all about.

9

DLM for relational databases

Given where our particular strengths lie, for this first edition of the book we’ll aim to discuss DLMin general terms (with an acknowledgement that we’ll generally be intending our information toapply to relational databases rather than NoSQL databases at this stage in the journey), and we’lluse SQL Server for specific examples because that’s what we know best. Of course, that all said, butwhere appropriate, we’ll mention equivalent tools for other relational systems

10

8. Version control for DLMWe’ve hinted at this earlier in the book, but one of the absolulutely fundamental practices youshould embrace, whether you intend to practice our flavor of Database Lifecycle Management ornow, is to get your database under version control. Not only does this make your life immenselyeasier when it comes time to audit, but you’ll find that it enables you to remove huge amountsof risk from your work, as well as keep a clear history of who has done what to the database. Withthat in mind, there are a few things you need to consider right away,

• What are you going to version (latest state vs every change) and how?• Versioning through your ORM – code first.• What are your branching strategies?

11

9. BuildTalking about possible approaches to “building” a database, as well as defining exactly what wemeanwhen we talk about a database “build”. Covering processes like manual build from VCS, ContinuousIntegration, and explaining the value of early delivery – doing builds more often and on smallchange-sets, and testing them to assess the design, overall approach and get advice from affectedteams. Finally Continuous Delivery where “chucking it over the wall” is part of an automatedprocess. Explaining how regular builds improve the visibility of your changes, ensure that your buildprocess is repeatable, and how a good build process is designed to be monitored and improved.

12

10. CI for DatabasesIntroduce the basic ideas of CI for the benefit of database folks who aren’t familiar with the term.Particularly the practice (and benefits) of making sure changes are integrated as regularly as possible.I’d expect this to make the requirement of good source control extremely clear, as well as outoutlining the scope and function of a build / CI server, the pipelines they rely on, and what they canand can’t do.

13

11. Testing DatabasesThis is pretty dense chapter, and it might need breaking in two. Start from the premise that manydevelops (arguably particularly database developers are not using a version control system thattheir changes are tracked in). Once we’ve hammer home the point about version control being anessential foundation, we can talk about test types: Smoke, integration, Unit, migration / deployment.They key is to move frommanually running testing to spot specific issues, towards creating a testingframework that enables a full suite of tests to be run automatically at some arbitrary trigger. It’llalso touch on test data, and how no database test is really effective with the right amounts of the rightkind of data, and strategies for acquiring or generating good test data. Finally, we’ll explain howeffective automated testing is a key enabler for rapid delivery and high quality / low risk databasedevelopment.

14

12. Release / DeploymentWe’ll start with the basics of what’s involved in a database deployment – testing your migrations,scripts and stored procedures to ensure there’s no data-loss as a result of your changes, ensuringthat the right permissions are in place, checking to make sure that the state of the target database(presumably production) hasn’t drifted. From there, we can look at more sophisticated techniques,like automating and instrumenting as much of your deployments as possible, being able toautomatically roll-back if your release encounters any errors, and being able to move your changesthrough multiple environments with a minimum of manual effort to managing configurationsand permissions. The goal here is to provide as much visibility as possible into each step of yourdeployment process, to remove as much human interaction as possible so that it is fully repeatable,and to understand it well enough to be able to make targeted, considered improvements to the speedand reliability of your releases.

15

13. Monitoring and FeedbackLots of teams will tell you about how they’ve set up Nagios and Graphite and New Relic to GatherAll The Data, but we’re not going to talk about that. Instead, now that you’ve got the basics ofa database deployment pipeline set up, we’ll talk about what elements of your database lifecyclemanagement you should be monitoring, and offer some guidance as to which metrics are going tomatter. We’ll talk about how sophisticated teams have built tight feedback looks into their process,and what they do to make sure they act on the data they gather.

16

14. Issues tracking for DLMIf issues aren’t being tracked in some kind of public channel, then they’re not visible, and thus theycan’t be acted upon. At the very least, you should be taking anything you discover needs action(because you’ve already started putting a good holistic monitoring system in place, right?), anddocumenting on post-it notes on your team backlog or similar. As your processes evolve, and youbecome more sophisticated, you’ll start looking at integrating an issue tracking system with yourplanning process, and eventually your version control system. We’ll look at the things you shouldbear in mind as you go through this process, how to make the transition as painlessly as possible,and how to make sure that the database and the application layers don’t quiet split into issue silos.

17

DLM in the Enterprise

We’ve covered the foundations of the DLM – the practices which are essential to buildinga development and delivery pipeline, and a good feedback system. To implement or build acomprehensive process for database (and application) lifecycle management, there are other keyelements to consider. More to the point, if you’re in a large / mature / regulated enterprise, thesepractices are going to be essential to getting the full benefit of DLM in your organization.

18

15. Configuration ManagementTo start with, all of your database configuration management and administration tasks will bemanual activities. As you become more comfortable with source control and the way your coredatabase lifecycle management processes are working, you should start to consider how to automate(or at least version control) as much of your work as possible. To start with, we’ll look at how tocapture all of your database objects and agent tasks, and ensure they are version controlled. Fromthere, we’ll look at what you can achieve with build scripts and access permission scripting,and how to set up effective dependency management so that you can manage your entire databaseconfiguration either entirely automatically, or at least from a single tool or command line.

19

16. Access Control / SecurityHaving touched on how automating access permissions will let you achieve completely repeatable,transparent and fast configuration changes to your databases, it’s essential that we take a deeperlook at access control and security in a world where you’re automating more and more of theday-to-day work around your databases. We’ll look at how your security considerations willhave changed, now that your database lifecycle management processes have evolved, and how tomake sure your database stays secure even as it evolves and improves.

20

17. Audit, Traceability, ComplianceAlthough audit and compliance are things you should always be considering, the increasedtransparency afforded byDLMmeans that we’re now in a position to build a truly comprehensiveand pain-free audit process. With much (if not all) of our database tracked in version control andlinked to an issue tracked, and database deployment pipeline instrumented and providing constantfeedback we have more on-demand visibility into the change and access history of our database thanever before. And because clear communication is a natural part of a healthy DLM process, everyonein the team is aware of their audit responsibilities.

21

18. Documenting DatabasesWe all like to believe that our code is self-evident, but when you’re implanting a sophisticatedDLM process, and you’re starting to work more closely with teams and functions you mighthistorically have avoided, it is crucial that your databases be well-documented. Borrowing fromwell recognized (though not necessarily clear cut) principles from application development, all theroutines and scripts that are a part of your database should include at least a note about intendedfunctionality. We’ll touch upon ways to automate the collection of data andmeta-data (includingboth the explicit notes from humans, and measures like code-quality metrics) and present it in someof themost effective ways possible. Improving your documentationwill encourage you to bemindfulof your code, as well as the way it is interacting with other services and libraries.

22

19. ReportingBasic reporting tends to come down to a kind of gut instinct – a sense of “Heavens, has it reallytaken that long for that process to run?” Accurate though that may be, it’s not always useful introuble-shooting or communicating with teams. As your DLM processes become more transparent,you’ll almost certainly be asked to describe their current state to stake-holders, so being able toautomatically track and document quality and health metrics for both your processes and thedatabases themselves will save you from a rather tedious task. Once you’ve got a consistent reportingprocess in place, we’ll also look at what sorts of trends it becomes valuable to track over time, andhow to respond to them.

23

The Data Ecosystem, and ExtendingDLM

By this point, we’ve described what we mean by ‘Database Lifecycle Management’, and it’s clearwhy it’s important to think about evolving the way we develop and deliver our relational databases.We’ve guided you through the process of setting up the core of your DLM pipeline, as well as thesurrounded elements of DLM processes that will lead to a sophisticated and seamless integrationbetween your application, database, and business.

The next step is to take a deep dive into some of the more design-centric aspects of DLM, as well ashighlighting how it will impact a few specific scenarios (ETL, your CMS etc) to illustrate the benefitsyou are working towards.

24

20. Hybrid database environmentsBefore we go on, it’s worth noting that we’ve not yet touched upon the wide variety of environmentsin which youmight find yourself trying to implement a modern DLM process. With that in mind, it’scertainly worth acknowledge that different types of data and process are likely to require differenttypes of database, and that the process of adopting DLMmay enable you (or even force you) to startconsidering a hybrid data environment.

In this section, we’ll briefly review the major types of database, what these are suitable for (at leastat a domain concept level, not a specific application or service level), and roughly outline how theTRIM principles could be applied in each case.

• Relational (for many years (1980s-2000s) this was the only real choice, and we’re still livingwith that dominance now)

• Object• Graph• Document• Wide column• Event store / log stream• Map-Reduce (Hadoop etc.)

25

21. Operational ModelsModern applications, and the databases that drive them, are run on a wide variety of operationalmodels, including on-premise, IaaS, PaaS, DBaaS, in-memory databases, virtual machines, andhybrid environments, to name just a few. To understand how DLM is going to play out in thosedifferentmodels, and how it’s going to impact theway you design your database, we’ll reviewthe main types of operational model available today, and the implications for different roles (Devs,Testers, DBAs, IT Ops people, etc.)

Just as an example, we might explore the point that, with a DBaaS (e.g. Azure SQL Database), youwould never want to couple requirements from two different client organizations together, and thatthe same model ought to occur within an organization too.

26

22. DLM for ETL SystemsFew organizations, when designing their OTLP systems, give much thought to the continuinglifecycle of the data, outside of that system. They don’t consider how they are going to transformand aggregate data from the various sources around the organization into a central data sourcethat, through data mining, can help support and drive their business decisions. It results in chaotic,overrunning ETL processes. We’ll explain how to TRIM them: - Identify and disentangle conflictingETL processes - Instrument ETL - Parallelize data extraction and processing - Impose data formatstandards and enforce data validation checks - Use Value Stream Mapping to highlight areas forimprovement

27

23. Content Management Systems(CMS)

With the rise of, amongst other things, Microsoft Dynamics, and the continued enterprise appetitefor SharePoint, how does DLM affect or work with a “black box” data store? How can you shipchanges and get relevantmonitoring data when the database you’re trying to improve is sealed away,relatively obfuscated, or otherwise obscured and only partially accessible? How can you improveit’s TRIM characteristics?

• What does DLM look like for a CMS developer or administrator?• What are the most effective and efficient ways to interact with the boiler-place databasebackend for common web-based things?

• How do you synchronize content from upstream or downstream environments on a regularbasis?

• How much of your interactions with a CMS backend can be usefully automated, and whatdata can you generally get access to?

28

24. Business Intelligence (BI)By far one of the most important scenarios we can discuss at this point is your Business Intelligence,or BI. With ever-increasing amounts of customer data being collected, and newer, more polishedtools emerging for BI specialists, the way that companies set themselves up to enable internalanalytics and intelligence has to be factored into their systems design and database architecture.We’ll look at a few of the things you should bear in mind if you haven’t already set off downthis path, as well as a few things you should consider doing right away, and a few anti-patterns youshould aim to avoid.

• What is different for BI developers?• The need to extract BI data to a data mart rather than directly from the transactional database• The organizational and technical tension that a poorly designed (monolithic) BI process canapply.

• How to design BI systems and processes for evolving databases

29

25. Data Retention, Discovery andCuration

A common side-effect of DLM is that the organization becomes much more aware of how and whyit is using customer’s data, and more sensitive to what data they can and should be retaining.Indeed, a growing movement called Datensparsamkeit (“Data sparse ness” or “Data saving ness”)recommends that we save as little as possible in order to fulfil the needs of the service/product weare proving to users.

Similarly, one possible result of a sophisticated DLM process is that your organization’s data isstored in a number of carefully chosen, fit-for-purpose database or services, and you now have achallenge when it comes to discovering, cataloguing, and coordinating or curating that internal datalandscape.

• How do we define retention policies across a heterogeneous database environment?• What are the processes and tools we can use for data discovery and audit?• What are some of the good practices for the coordination of data references (e.g. OrderID)across different teams and program streams?

30

Appendix

31