Post on 26-Mar-2015
®
IBM Software Group
© 2006 IBM Corporation
Agility in the Database:Data Doesn’t Have to be a Four-Letter Word Any More
Scott W. AmblerPractice Leader Agile Development
IBM Software Group | Rational software
Scott Ambler - Background
Practice Leader Agile Development
Fellow – International Association of Software Architects
www-306.ibm.com/software/rational/bios/ambler.html
IBM Software Group | Rational software
Presentation Overview
Warning!
Critical Observations
Adoption Rate
Evolutionary/Agile Data Modeling
Database Refactoring
Database Testing
Other Important Techniques
Conclusion
IBM Software Group | Rational software
Warning!
I’m spectacularly blunt at times
Many new ideas will be presented
Some may not fit well into your existing environment
Some will challenge your existing notions about software development
Some will confirm your unvoiced suspicions
Don’t make any “career-ending moves”
Be skeptical but open minded
IBM Software Group | Rational software
Critical ObservationsModern Development Leading development lifecycles are
iterative and incremental
New applications typically use a combination of object and data technologies – the world isn’t pure
There is growing proof that serial/traditional approaches are ineffective
Data in only one of many important aspects, and it is rarely the primary one
There is growing evidence that craftspeople, or “generalizing specialists”, are significantly more effective than specialists
Traditional IT organizations appear to be vulnerable to offshore outsourcers
Cultural There is a serious rift between the data and
development communities Each community is relatively ignorant of
what the other one is doing There are few books within either
community that do the other justice The agile movement is coming from the
development community Many developers are prepared to ignore
data professionals, and do so quite often
IBM Software Group | Rational software
66% of Development Teams Go Around Data Groups: Why?
8%
17%
20%
36%
19%Don't Know it Exists
Don't Know TheyShould Work WithThem
Data Group TooDifficult
Data Group Too Slow
Data Group OffersLittle Value
Source: www.ambysoft.com/surveys/
IBM Software Group | Rational software
Adoption Rate of Agile Database Techniques
Innovators Early Adopters Early Majority Late Majority Laggards
The“Chasm”
®
IBM Software Group
© 2006 IBM Corporation
Evolutionary/Agile Data Modeling
What is Evolutionary/Agile Data Modeling
An Example
Why This Works
IBM Software Group | Rational software
Agile Data Modeling
Data modeling is the act of exploring data-oriented structures.
Evolutionary data modeling is data modeling performed in an iterative and incremental manner.
Agile data modeling is evolutionary data modeling done in a collaborative manner.
Data modeling is only one of many aspects of modeling, see www.agilemodeling.com/artifacts
See Agile Model Driven Development (AMDD) www.agilemodeling.com/essays/amdd.htm
IBM Software Group | Rational software
Karate School Management System Example
Based on an actual organization
Did the same problem two ways:
Without an initial conceptual model www.ddj.com – A three-issue series
With an initial conceptual modelwww.agiledata.org/essays/agileDataModeling.html
IBM Software Group | Rational software
The Initial Requirements(A Karate School Management System)
Iteration User Stories
1 Maintain student contact information
Enroll student
Drop student
Record payment
2 Promote student to higher belt
Invite student to grading
Email membership to student
Print membership for student
3 Schedule gradings
Print certificate
Put membership on hold
4 Maintain product information
Sell product
5 Print catalog of products
Order product for inventory
Order product for student
6 Organize tournament
Enroll participant in tournament
Send out tournament announcement email to past participants
Print tournament announcement letters to past participants
IBM Software Group | Rational software
Initial Domain Model
IBM Software Group | Rational software
Iteration 1 PDM
IBM Software Group | Rational software
Iteration 2 PDM
IBM Software Group | Rational software
Iteration 3 PDM
IBM Software Group | Rational software
Updated Requirements
Iter. User Stories
4 Enroll child student
Offer family membership plan
Support child belt system
5 Enroll student in Tai Chi
Support Tai Chi belt system
Enroll student in cardio kick boxing
Support Tai Chi belt system
Support the belt order for each style
6 Maintain product information
Sell products
7 Print catalog of products
Order product for inventory
Order product for student
8 Organize internal special event (special classes, internal tournaments, …)
Enroll student in special event
Print special event certificate for student
IBM Software Group | Rational software
Updated Domain Model
IBM Software Group | Rational software
Iteration 4 PDM
IBM Software Group | Rational software
Iteration 5 PDM
IBM Software Group | Rational software
Iteration 6PDM
IBM Software Group | Rational software
Why This Works
Up front, high-level modeling defines the landscape
JIT model storming enables:You to ask more intelligent questions because you know more about the
domain the longer you work in itStakeholders to answer more intelligently because they understand the
solution space betterFocus on functionality that is actually needed, not what might be needed
®
IBM Software Group
© 2006 IBM Corporation
Refactoring Databases
Database RefactoringWhy it’s HardThe ProcessExamplesWhy This Works
IBM Software Group | Rational software
Uncomfortable Question
How long would it take you to safely rename a column in a production database table?
Could you do it in a single day?
IBM Software Group | Rational software
Survey: How Long to Rename a Column in a Production Database?
25%
28%
21%
11%
7%
8%
Single Day
Within a Week
Within a Month
Within 3 Months
> 3 Months
Too Risky To Try
Source: www.ambysoft.com/surveys/
IBM Software Group | Rational software
0
5
10
15
20
25
30
35
40
1 to 10 11 to50
51 to100
101 to500
501-1000
1000+
One Day
One Week
One Month
Three Months
> Three Months
Too Risky
Length of Time to Rename Production Column (% Respondents)
IBM Software Group | Rational software
61.9% of Respondents Have Production Data Problems:Their Strategy to Address Data Source Problems
18%
33%
8%
33%
8%
No Strategy
Don't Make it Worse
Big Bang
Evolutionary
Other
Source: www.ambysoft.com/surveys/
IBM Software Group | Rational software
Database Refactoring
A database refactoring is a simple change to a database schema that improves its design while retaining both its behavioral and informational semantics.
A database schema includes both structural aspects such as table and view definitions as well as functional aspects such as stored procedures and triggers.
Important: Database refactorings are a subset of schema transformations, but they do not add functionality.
www.agiledata.org/essays/databaseRefactoring.html
IBM Software Group | Rational software
Why DB Refactoring is Hard
YourDatabase
YourApplication
OtherDatabases
OtherApplications
You Know About
OtherApplicationsYou Don't
Know About
DataFile
DataExtracts
DataFile
DataImports
PersistenceFrameworks
TestCode
IBM Software Group | Rational software
The Process of Database Refactoring
Verify that a refactoring is
needed
Run the tests
Change your schema
MigrateData
(optional)
[Fail]
[Pass]
[Not Needed]
[Finished]
[Work continues]
Choose theRight
Refactoring
Write aUnit Test
Deprecate the OriginalSchema
(optional)
Update ExternalAccess
Programs
Run the tests
AnnounceThe Refactoring
VersionControl Your
Work
[Fail]
IBM Software Group | Rational software
Rename Column
Customer <<Table>>
CustomerID <<PK>>Fname
Customer <<Table>>
CustomerID <<PK>>Fname { drop date = November 14 2006 }FirstName
SynchronizeFirstName { event = update | insert, drop date = November 14 2006 }
Customer <<Table>>
CustomerID <<PK>>FirstName
Original Schema
Transition Period
Resulting Schema
IBM Software Group | Rational software
Example:Replace Column
Original Schema
Transition Period
Resulting Schema
Customer
CustomerPOID <<PK>>CustomerNumber: integerFirstNameLastName
Customer
CustomerPOID <<PK>>CustomerNumber: integer { drop date = June 14 2006 }CustomerID: char(12)FirstNameLastName
SynchronizeCustomerIDNumber { event = update | insert, drop date = June 14 2006 }
Customer
CustomerPOID <<PK>>CustomerID: char(12)FirstNameLastName
IBM Software Group | Rational software
Example:RemoveView
Account
Original Schema
Transition Period
Resulting Schema
Customer
AccountDetails <<View>>
Account
Customer
AccountDetails <<View>>{ drop date = June 14 2006 }
Account
Customer
IBM Software Group | Rational software
Example:Merge Columns
Original Schema
Transition Period
Resulting Schema
Customer
PhoneCountryCodePhoneAreaCode { drop date = December 14 2006 }PhoneLocal { drop date = December 14 2006 }PhoneNumber
SynchronizePhoneNumber { event = update | insert, drop date = December 14 2006 }
Customer
PhoneCountryCodePhoneNumber
Customer
PhoneCountryCodePhoneAreaCodePhoneLocal
IBM Software Group | Rational software
Why This Works
The database schema is evolved in very small steps in a low-risk manner
Each refactoring is simple to implement
It enables an evolutionary approach while supporting the existing infrastructure
®
IBM Software Group
© 2006 IBM Corporation
Database Regression Testing
Database Regression Testing
Possible Things to Test
Current Database Testing
Add a test
Run the tests
Make a littlechange
Run the tests
[Fail]
[Fail]
[Pass]
[Developmentstops]
[Developmentcontinues]
IBM Software Group | Rational software
Current State of Database Regression Testing (July 2006)
95.7% of organizations considered data to be a corporate asset Of those, only 40.3% had a regression test suite in place
Of those, Only 63.3% allowed developers to run this test suite when they needed to
Of the organizations that didn’t have a test suite, only 31.6% had discussed putting one in place
63.7% of organizations implemented mission-critical functionality in the database Of those, only 46% had a regression test suite in place
Of those, Only 66.3% allowed developers to run this test suite when they needed to
Of the organizations that didn’t have a test suite, only 38.6% had discussed putting one in place
IBM Software Group | Rational software
Database Regression Testing
Agilists develop regression unit and acceptance test suites for their applications, why not for databases too?
You must test:Within the database (clear box)At the interface to the database (black box)
You must be able to put the database into a known state, therefore you need test data (generation)
IBM Software Group | Rational software
Things to Test For
IBM Software Group | Rational software
Do You Do Database Testing? (September 2006)
Yes66%
Inadequate Resources
20%
Never Considered
7%
Don't Know7%
IBM Software Group | Rational software
35.3
12.9
43.3
43.5
29.9
19.8
App Dev (TDD)
Data Warehouse(TDD)
End of DevIteration
End of DevLifecycle
During DataExtract
Don't Know
When Are You Testing the Database? (% Respondents)
IBM Software Group | Rational software
Types of Database Testing (% Respondents)
73.5
52.3
25.7
51.7
33.8
39.5
42.7
39.5
DB Input
DB Output
O/R Mappings
Stored Procs
View Definitions
RI Constraints
Column Constraints
Default Column Values
IBM Software Group | Rational software
Other Important Techniques
Configuration management of all project artifacts
Continuous integration
Encapsulate database access
Train developers in data-oriented skills
Train data professionals in development skills
Promote the concept of generalizing specialists www.agilemodeling.com/essays/generalizingSpecialists.htm
IBM Software Group | Rational software
Conclusion
Evolutionary development is the norm, not the exception
Agile development techniques are quickly being adopted
There is no reason why data professionals cannot work in an evolutionary, or even agile, manner
The software world has changed. Have you?
Visit www.agiledata.org
®
IBM Software Group
© 2006 IBM Corporation
Keep In Touch!
Scott W. Amblerwww-306.ibm.com/software/rational/bios/ambler.html
IBM Software Group | Rational software
References and Recommended Reading www.agilealliance.com www.agilemodeling.com www.agiledata.org www.ambysoft.com www.databaserefactoring.com www.enterpriseunifiedprocess.com Ambler, S.W. (2002). Agile Modeling: Effective Practices for XP and the UP.
New York: John Wiley & Sons. Ambler, S.W. (2003). Agile Database Techniques. New York: John Wiley &
Sons. Ambler, S.W. (2004). The Object Primer 3rd Edition: AMDD with UML 2. New
York: Cambridge University Press. Ambler, S.W. and Sadalage, P.J. (2006). Refactoring Databases: Evolutionary
Database Design. Reading, MA: Addison Wesley Longman, Inc. Larman, C. (2004). Agile and Iterative Development: A Manager’s Guide.
Reading, MA: Addison Wesley McGovern, J., Ambler, S.W., Stevens, M., Linn, J., Sharan, V., & Jo, E. (2003).
The Practical Guide to Enterprise Architecture. Prentice Hall PTR.