Kscope 2013 delphix
-
Upload
kyle-hailey -
Category
Technology
-
view
108 -
download
1
description
Transcript of Kscope 2013 delphix
Database Virtualization and
Instant CloningKyle Hailey
http://dboptimizer.com
Database Cloning Challenge
Business want data now.
Business don’t understand DBAs.
Databases getting bigger & harder to copy.
Developers want more copies.
Reporting wants more copies.
Everyone has storage constraints.
If you can’t satisfy the business demands your process is broken.
Two Parts
I. Cloning TechnologyII. Accelerate your business
Part I : Cloning Technology
3. Virtual2. Thin Provision1. Physical
=
database
1. Physical Cloning
Problem
Developers
QA and UAT
Reports
First copy
Production
• CERN - European Organization for Nuclear Research
• 145 TB database• 75 TB growth each year• Dozens of developers want copies.
workaroundsDevelopers
QA and UAT
ReportsShared
Sub set copy
Production
Many copies
Physical Clones
Database SubsetsShared Databases
Subsets
ProductionThe Production ‘Wall’
Classic problem is that queries that run fast on subsets hit the wall in production.
Developers are unable to test against all data
Shared Full
Shared access = Poor Productivity
Developers and tester get frustrated
Databases become old and unrepresentative of production.
Requires complex scheduling and management
Never enough environments
Average customer makes 12 copies of production- Charles Garry, Database Product Manager Oracle
Physical CopiesTime consuming
Time to make copies, days to weeksRMAN backup, archive logs, copy data over, recover
Meetings , days to weeksAdmins: System, Storage ,Database ,Network, manager coordination
Space consuming 40 devs x 2.5TB production = 100TB20 report DBs x 40 TB = 800TB
=> bottlenecks
Setup Develop
Setup
Setup Develop
QA
$40M
$75M
$850M
$27,000M
Storage
IT
Develop
Business
ERP Project Failures 2011
• NYC CityTime : delays $63 M => $760 M • Montclair Uni: delays sues PeopleSoft• Idaho : delays ERP cost millions
Standish : IT Project Failure Rate
1994 1996 1998 2000 2002 2004 2009
31% 40% 28% 23% 15% 18% 24%
★http://www.galorath.com/wp/software-project-failure-costs-billions-better-estimation-planning-can-help.php*http://www.pcworld.com/article/246647/10_biggest_erp_software_failures_of_2011.html
Clone 1 Clone 3
99% of blocks are Identical
Clone 2
2. Thin Provision Cloning
Clone 1 Clone 2 Clone 3
Thin Provision
2. Thin Provision Cloning
Cornerstone Technology: File System Snapshots
Thin Cloning
• Snapshot DB Files @ point in time• Copy off production• Snapshot again• Export over NFS or FC to Host• Recovery Database
Netapp tr-3761.pdf
Netapp
Snap Manager
SnapManagerRepository
Protection Manager
Snap Drive
Snap Manager
Snap Mirror
Flex Clone
RMANRepository
Production
Development
DBA
Storage Admin
NetApp Filer - DevelopmentNetApp Filer - Production
Production Database
Database Luns
Target A
Target B
Target C
Clone 1
Clone 2
Clone 3
Clone 4
Snap mirror
Snapshot Manager for Oracle
Flexclone
Repository Database
Netapp
SnapDrive
Protection Manage
2. Thin Provision Cloning
2. Thin Provision Cloning
3. Database Virtualization
Virtualization Layer
38
Virtualization
DatabaseVirtualizationAppliance(DVA)
3 Clones Physical 3 Clones Virtual
Install Delphix on x86 hardware
x86 hardware
Allocate Storage to Delphix
Allocate StorageAny type
One time backup of source database
Database
Production
Instance
File system
RMAN APIs
Delphix Compress Data
Database
Production
Instance
File system
Data is compressed typically 1/3 size
Incremental forever change collection
Database
Production
Instance
File system
Changes are collected automatically foreverData older than retention widow freed
Typical Architecture
Database
File system
Production
Instance
Database
File system
Development
Instance
Database
File system
QA
Instance
Database
UAT
Instance
File system
Clones share duplicate blocks
Development
Instance
Database
Production
Instance
File system
vDatabase
QA
Instance
UAT
Instance
vDatabase vDatabase
Source Database Clone Copies of Source Database
Use Cases
1. Development2. Recovery3. Reporting
1. Development Acceleration
1: Development Acceleration
a) Developer each get a copy– Fast, fresh, full, frequent– Self service
b) Branchingc) Federated
Source
FastSource Database
Target HostVirtual
Database
NFS
Fiber
Fiber
RMAN over TCP
No Data Movement
Source
Fresh
Virtual Database
Fiber
Source
Frequent
Virtual Database
Virtual Database
Target Hosts
Virtual Database
Virtual Database
Fiber
Full clones
Self Service
1 b) Branching and Rapid QA
dSource
1 b) Branching
Developer VDB
QA VDB
Devv2.6 v2.6v2.6
QA UAT
v2.6
v2.6 v2.6v2.6v2.7
v2.6 v2.6v2.6v2.8
v2.6v2.6 v2.6v2.6
v2.6v2.7 v2.6v2.7
v2.6v2.8 v2.6v2.8
Devv2.6 v2.6v2.6
QA UAT
v2.6Production
v2.6 v2.6v2.6v2.7
v2.6 v2.6v2.6v2.8
Source Control for the database data
v2.6v2.6 v2.6v2.6
v2.6v2.7 v2.6v2.7
v2.6v2.8 v2.6v2.8
DevProd
2.6
Dev
QA
Prod
2.6
Dev
QA
UAT
Prod
2.6
Dev
QA
UAT
Prod
Dev
QA
UAT
2.6
2.7
Dev
QA
UAT
Prod
Dev
QA
UAT
2.6
2.7
Dev
QA
UAT2.8
Dev
QA
UAT
Prod
Dev
QA
UAT
2.6
2.7
Dev
QA
UAT2.8
Data Control = Source Control for the Database
Dev
QA
UAT
Dev
QA
UAT
2.6
2.7
Dev
QA
UAT
2.8
Data Control = Source Control for the Database
Production Time Flow
1 c) Federated Cloning
Source2
Source3
Source1
1 c) Federated sources
Virtual Database
Virtual Database
Virtual Database
Virtual Database
“I looked like a hero”Tony Young, CIO Informatica
1. Review Development Use Cases
a) Developer each get a copy
b) QA
c) Federated
2. Recovery, Testing, Forensics
a) Forensicsb) A/B testingc) Recovery
Source
2 a) Forensic Analysis
Virtual Database
Source
2 b) Upgrades, Patches, RAT, A/B
Virtual Database
• Production vs Virtual– invisible index on Prod– Creating index on virtual
• Flashback vs Virtual• Keep tests for compare
2 b) Upgrades, Patches, RAT, A/B
2 c) Recovery
Source
2 c) Logical Recovery Production
Virtual Database
Source
2 c) Logical Recovery Development
Virtual Database
VDB rolled back
Source
Recovery
VDB
V2P
2. Recovery, Testing, Forensics
a) Forensics
b) A/B testing
c) Recovery : Logical and physical
3: reportinga) Fast refreshes
b) Temporal queries
c) Confidence testing
Fast Refreshes
• Refresh in minutes• Without data movement• Faster , cheaper
Temporal Data
3: reportinga) Fast refreshes
b) Temporal queries
c) Confidence testing
Review: Use Cases
1. Developmenta) Full, Fresh, Fast , Self Serveb) Branchingc) Federated
2. Recovery, Testing :a) Forensicsb) Testing : A/B, upgrade, patchc) Recovery: logical, physical
3. Reportinga) Fast refreshb) Temporal Datac) Confidence testing
over 10 times
"perhaps the single largest storage consolidation opportunity history“
Oracle 12c
80MB buffer cache ?
200GBCache
5000
Tnxs
/ m
inLa
tenc
y
300 ms
1 5 10 20 30 60 100 200
with
1 5 10 20 30 60 100 200Users
8000
Tnxs
/ m
inLa
tenc
y
600 ms
1 5 10 20 30 60 100 200Users
1 5 10 20 30 60 100 200
Database Virtualization