Evolutionary Database Development

Software Development Process is Changing

Fix / Integrate $

Test

Code

DesignSpecifications

Use Cases / Functional Specs

Requirements Gathering

Project Plan/Estimation

$

$

$

$

Level Stories

Vision & High-

Iteration

Release

© ThoughtWorks 2009

Team Collaboration

DBDB

AppsApps

BackEndBackEnd

DEVSI’m responsible for Web design I’m responsible for Web design

DBA

I’m responsible for Database

I’m responsible for Database

Domain expert

I’m responsible for backend

apps

I’m responsible for backend

apps

Traditional Team

DBA

Our responsible for business value

Our responsible for business value

Our responsible for business value

Our responsible for business value

It’s our responsibility

It’s our responsibility

DEVS

Domain Expert

Agile Team

In Agile team, DBA

•Role!= Person

•knowledge of the functionality

•Acknowledge team in interaction

• DBA sit close to all the roles

• Help make a decision

• Educating Developers write better SQL

• How to make the DBA redundant

Best Practices

Refactoring

“A disciplined way to make change to your source code to improve its design, making it easier to work with”

Martin Folwer

CustomerCustomer

…Balance…Balance

AccountAccount

AccountID(PK)AccountID(PK)

CustomerCustomer

……

AccountAccount

AccountID(PK)Balance

AccountID(PK)Balance

Before:Before: After:After: Behavioral semantics

•Change structure

A small change to Database schema that improves its design while retaining both its behavioral and informational semantics

•Data Migration•Verify data quality

Informational semantics

Database Refactoring

• Structural• Data Quality• Referential Integrity• Architectural• Method• Non-Refactoring Transformation

What will be changed in DB Refactoring?

• Small Steps• Frequent Changes• Test First

Principles

• Test your Database Schema• Test the way Apps uses schema• Validate your data migration• Test external program code• Check data quality• …

TEST

Fail Fast

Fail Often

Test-Driven is also important for DB design

• Not trying to "Get it Right up Front“

• Build the simplest thing that can possibly work

• Treat changes as database refactoring ... Every DAY

• functionality added in increments

Evolutionary Modeling Design

A Story

Background

CRM system•Transactional operations •Reporting/ Statistics functions for manager•Based on Legacy system •24*7 , a very busy system

Client’s expectation•Improvement of Usability and performance

Legacy Database•Sql server 2005•200G

Team•Distributed Agile team. (Beijing + Hongkong)

Refactoring database schema

Introduce transition period for SAFTY

Never get rid of old schema immediately

Data sync in real time

Legacy DBLegacy DB NEW DBNEW DB

What we did in this project

FlatFileFlatFile

systemNamesystemName

SystemNameID(PK)SystemLogin

…

SystemNameID(PK)SystemLogin

…

systemName_logsystemName_log

SystemNameID(PK)OperDatetime

SystemNameID(PK)OperDatetime

TrigerTriger AccountAccount

AccountID(PK)AccountName

SystemNameIDIsdelete

AccountID(PK)AccountName

SystemNameIDIsdelete

SystemNameIDNewvalueOldValue

columnName

SystemNameIDNewvalueOldValue

columnName

Read DataRead Data

Near real Time

Read DataRead Data

Onetime Migration

Need to figure out logical mappingProblem:1.There are over 400 procs, over 1000 tables, over 150 views2.Data Concept/logics changed

How we resolve:1.Communication is very important2.Split a function into small parts, more smaller more easy more safe

PersonPersonOrganisationOrganisation

PersonPersonUserUser

contactcontact

 The process of database refactoring

Verify data quality by domain logics after one time migration

UserUser

UserID(PK)FirstNameLastName

…

UserID(PK)FirstNameLastName

…

1.One user Should not have 2 office telephoneSelect userFrom user a left join telephone bOn a.userid=b.useridGroup by a.userHaving count(*)>=2

2. A telephone should be assigned to a userSelect telephoneFrom telephone Where userid is null

3. A telehpone shoulde not be assigned to 2 or more userSelect officeTeleFrom telephone a left join userGroup by officeTeleHaving count(userid)>=2

… adding more verification sql

TelephoneTelephone

TeleID(PK)OfficeTeleHomeTeleUserID(FK)

TeleID(PK)OfficeTeleHomeTeleUserID(FK)

Sanity Check

1. Brainstorming2. Feedback from CI/Production

Don’t make an issue to happen for tw

o times

Put test scripts into CIPut verification scripts into CI

Run verification scripts in production often, be aware of production data

ProductionUsers

QA

Production

AppApp

Pre-Production

AppApp

Devs

Highly iterative

Development System and

Acceptance Testing

Operations and support

Devs’

AppApp

DB

AppApp

Bug ReportsBug ReportsBug ReportsBug Reports

Bug ReportsBug Reports

Project-level Integration

Testing

Frequent Deployment

Frequent Deployment

Frequent Deployment

Frequent Deployment

Highly Controled

Deployment

Highly Controled

Deployment

Our workflow

CVS

• Tests• SanityCheck

• Tests• SanityCheck

Brainstorming

Brainstorming

Devs

The strategy for migration performance

1. Online system don’t allow that migration spend too long time

ContactInfoContactInfo

Messid(pk)C1C2…

Messid(pk)C1C2…

ContactNoteContactNote

ContactNoteId(pk)ContactNote

ContactIdMessId

ContactNoteId(pk)ContactNote

ContactIdMessId

MigrationMigration

Update contactNote aSet ContactNote=c1,ContactId=c2From ContactInfo where a.messid=b.Messid

Too long

Delcare row_num number:=0;Begin for c_CN in (select MessId,C1,C2 from contactInfo a left join CI_Mig_log b on a.Messid=b.messid where flag=0) update contactNote set contactNote=c_CN.c1, contactId=c_CN.c2 where Messid=c_CN.Messid update CI_MIG_log f set f.flag=1 where messid=c_Cn.MessId row_num:=row_num+1; if mod(row_num,1000)=0 then commit; end if;End loop;

Cancel it if there is no enough resource for migration

ContactInfoContactInfo

Messid(pk)C1C2…

Messid(pk)C1C2…

ContactNoteContactNote

ContactNoteId(pk)ContactNote

ContactIdMessId

ContactNoteId(pk)ContactNote

ContactIdMessId

MigrationMigration

CI_Mig_logCI_Mig_log

MessID(PK)Flag

MessID(PK)Flag

Versioning Database• All database schemas can be thought of as DB refactoring• As updates are applied to a database, the changes will be

recorded in a table similar to the following:

Change Date

1_Create_Customer_Table.sql 4-15-07

2_Add_e-mail_address_column.sql 4-17-07

3_Add_fax_number_column.sql 4-18-07

4_Add_transaction_table.sql 4-21-07

5_Add_transaction_status_column.sql 4-24-07

6_Add_customer-transaction_view.sql 4-27-07

Put them under configuration management control

CREATE TABLE money ( eek NUMBER);

//Test for…

DB

DDL

Insert into AA(mydata)Values(11);

Meta Data

Delete from ..

DML

Create index optimization

Merge intoData Migration

Tests

Installationscripts

Configuration of DB project

Database project under version control

Tiny db backup

Deltascripts of dbdeploy

For defining db objects which depends on schema

For data onetime migration

For data sync

For checking dirty data

Tool exclusively for database project

other scripts…

Database Deployment

An automated processAn automated process is needed to make the process is needed to make the process of upgrading out-of-date databasesof upgrading out-of-date databases

Our Practices

• Nothing is used only once• Automate tasks such as

• Physical table deployment• Usage statistics• Schema verification• Data migration verification

• Introduce tools ,like• Ant, dbdeploy

DB Deploy - http://dbdeploy.com/

Management DB deploymnet

DBDeploy

http://dbdeploy.com/documentation/getting-started/rules-for-using-dbdeploy/

• Naming convention for delta scripts:NUMBER COMMENT.SQLe.g. 1_Create_Customer_Table.sql…

• Undo section – marked by comments:

CREATE TABLE FOO ( FOO_ID INTEGER NOT NULL, FOO_VALUE VARCHAR(30) );

ALTER TABLE FOO ADD CONSTRAINT PK_FOO PRIMARY KEY (FOO_ID);--//@UNDO DROP TABLE FOO;

<target name="gen-and-exec-delta-script">

<dbdeploydriver="oracle.jdbc.OracleDriver"

url="jdbc:oracle:thin:@localhost:1521:XE"

userid="dylan"

password="nalyd"

dir="./sql/deltas/"

outputfile="./build_output/db-deltas-hsql.sql"

dbms="ora"/>

<sql

driver=" oracle.jdbc.OracleDriver"

url=" jdbc:oracle:thin:@localhost:1521:XE "

userid="dylan"

password="nalyd"

src="./build_output/db-deltas.sql"

onerror="abort"/>

</target>

Ant

DBDeploy

1. Go to directory with SQL files:1. “1 create_customer_table.sql”2. “2 add_customer_id_constraint.sql”

2. Run “ant”

Output:gen-and-exec-delta-script:[dbdeploy] dbdeploy v2.11[dbdeploy] Reading change scripts from directory C:\Projects\dbdeploy-demo\sql\deltas...[dbdeploy] Changes currently applied to database:[dbdeploy] 1, 2[dbdeploy] Scripts available:[dbdeploy] 1, 2, 3, 4[dbdeploy] To be applied:[dbdeploy] 3, 4[sql] Executing file: C:\Projects\dbdeploy-demo\build_output\db-deltas.sql[sql] 8 of 8 SQL statements executed successfully

Automate Tasks

•CreateNewTestDB

•upgradeDB

<target name= “-parseDbScxripts” >…</target>…

<target name="-upgradeDB" depends="-parseDbScripts, -dbdeploy, -runDeltaScript, -dropDbLogic, -createDbLogic" description="Upgrade specified database to latest version" />

<target name="rebuildDB" depends=" -parseDbScripts, -dropDb, -createDb, -initialiseDb, -dbdeploy, -runDeltaScript, -createDbLogic" description="drop, recreate and initialise the Connect database" />

Batch file:library\nant\nant.exe –buildfile:evovle.build –D:rebuildDB -logfile:build.txt

Shared to devs

Do what we want by just One Command

•Reference:•Evolutionary Database Designhttp://martinfowler.com/articles/evodb.html

•Refactoring Databases: Evolutionary Database Design

Q&A