Kscope 2013 delphix

96
Database Virtualization and Instant Cloning Kyle Hailey http://dboptimizer.com

description

Delphix Database Virtualization

Transcript of Kscope 2013 delphix

Page 1: Kscope 2013 delphix

Database Virtualization and

Instant CloningKyle Hailey

http://dboptimizer.com

Page 2: Kscope 2013 delphix

Database Cloning Challenge

Business want data now.

Business don’t understand DBAs.

Databases getting bigger & harder to copy.

Developers want more copies.

Reporting wants more copies.

Everyone has storage constraints.

If you can’t satisfy the business demands your process is broken.

Page 3: Kscope 2013 delphix

Two Parts

I. Cloning TechnologyII. Accelerate your business

Page 4: Kscope 2013 delphix

Part I : Cloning Technology

3. Virtual2. Thin Provision1. Physical

Page 5: Kscope 2013 delphix

=

database

1. Physical Cloning

Page 6: Kscope 2013 delphix

Problem

Developers

QA and UAT

Reports

First copy

Production

• CERN -  European Organization for Nuclear Research

• 145 TB database• 75 TB growth each year• Dozens of developers want copies.

Page 7: Kscope 2013 delphix

workaroundsDevelopers

QA and UAT

ReportsShared

Sub set copy

Production

Many copies

Page 8: Kscope 2013 delphix

Physical Clones

Database SubsetsShared Databases

Page 9: Kscope 2013 delphix

Subsets

Page 10: Kscope 2013 delphix

ProductionThe Production ‘Wall’

Classic problem is that queries that run fast on subsets hit the wall in production.

Developers are unable to test against all data

Page 11: Kscope 2013 delphix

Shared Full

Page 12: Kscope 2013 delphix

Shared access = Poor Productivity

Developers and tester get frustrated

Databases become old and unrepresentative of production.

Requires complex scheduling and management

Page 13: Kscope 2013 delphix

Never enough environments

Average customer makes 12 copies of production- Charles Garry, Database Product Manager Oracle

Page 14: Kscope 2013 delphix

Physical CopiesTime consuming

Time to make copies, days to weeksRMAN backup, archive logs, copy data over, recover

Meetings , days to weeksAdmins: System, Storage ,Database ,Network, manager coordination

Space consuming 40 devs x 2.5TB production = 100TB20 report DBs x 40 TB = 800TB

=> bottlenecks

Page 15: Kscope 2013 delphix
Page 16: Kscope 2013 delphix

Setup Develop

Page 17: Kscope 2013 delphix

Setup

Page 18: Kscope 2013 delphix

Setup Develop

QA

Page 19: Kscope 2013 delphix

$40M

$75M

$850M

$27,000M

Storage

IT

Develop

Business

Page 20: Kscope 2013 delphix
Page 21: Kscope 2013 delphix

ERP Project Failures 2011

• NYC CityTime : delays $63 M => $760 M • Montclair Uni: delays sues PeopleSoft• Idaho : delays ERP cost millions

Standish : IT Project Failure Rate

1994 1996 1998 2000 2002 2004 2009

31% 40% 28% 23% 15% 18% 24%

★http://www.galorath.com/wp/software-project-failure-costs-billions-better-estimation-planning-can-help.php*http://www.pcworld.com/article/246647/10_biggest_erp_software_failures_of_2011.html

Page 22: Kscope 2013 delphix
Page 23: Kscope 2013 delphix

Clone 1 Clone 3

99% of blocks are Identical

Clone 2

Page 24: Kscope 2013 delphix
Page 25: Kscope 2013 delphix

2. Thin Provision Cloning

Page 26: Kscope 2013 delphix

Clone 1 Clone 2 Clone 3

Thin Provision

Page 27: Kscope 2013 delphix

2. Thin Provision Cloning

Cornerstone Technology: File System Snapshots

Page 28: Kscope 2013 delphix

Thin Cloning

• Snapshot DB Files @ point in time• Copy off production• Snapshot again• Export over NFS or FC to Host• Recovery Database

Page 30: Kscope 2013 delphix

Netapp

Snap Manager

SnapManagerRepository

Protection Manager

Snap Drive

Snap Manager

Snap Mirror

Flex Clone

RMANRepository

Production

Development

DBA

Storage Admin

Page 31: Kscope 2013 delphix

NetApp Filer - DevelopmentNetApp Filer - Production

Production Database

Database Luns

Target A

Target B

Target C

Clone 1

Clone 2

Clone 3

Clone 4

Snap mirror

Snapshot Manager for Oracle

Flexclone

Repository Database

Netapp

SnapDrive

Protection Manage

Page 32: Kscope 2013 delphix
Page 33: Kscope 2013 delphix

2. Thin Provision Cloning

Page 34: Kscope 2013 delphix

2. Thin Provision Cloning

Page 35: Kscope 2013 delphix
Page 36: Kscope 2013 delphix

3. Database Virtualization

Page 37: Kscope 2013 delphix
Page 38: Kscope 2013 delphix

Virtualization Layer

38

Virtualization

Page 39: Kscope 2013 delphix

DatabaseVirtualizationAppliance(DVA)

3 Clones Physical 3 Clones Virtual

Page 40: Kscope 2013 delphix
Page 41: Kscope 2013 delphix

Install Delphix on x86 hardware

x86 hardware

Page 42: Kscope 2013 delphix

Allocate Storage to Delphix

Allocate StorageAny type

Page 43: Kscope 2013 delphix

One time backup of source database

Database

Production

Instance

File system

RMAN APIs

Page 44: Kscope 2013 delphix

Delphix Compress Data

Database

Production

Instance

File system

Data is compressed typically 1/3 size

Page 45: Kscope 2013 delphix

Incremental forever change collection

Database

Production

Instance

File system

Changes are collected automatically foreverData older than retention widow freed

Page 46: Kscope 2013 delphix

Typical Architecture

Database

File system

Production

Instance

Database

File system

Development

Instance

Database

File system

QA

Instance

Database

UAT

Instance

File system

Page 47: Kscope 2013 delphix

Clones share duplicate blocks

Development

Instance

Database

Production

Instance

File system

vDatabase

QA

Instance

UAT

Instance

vDatabase vDatabase

Source Database Clone Copies of Source Database

Page 48: Kscope 2013 delphix

Use Cases

1. Development2. Recovery3. Reporting

Page 49: Kscope 2013 delphix

1. Development Acceleration

Page 50: Kscope 2013 delphix

1: Development Acceleration

a) Developer each get a copy– Fast, fresh, full, frequent– Self service

b) Branchingc) Federated

Page 51: Kscope 2013 delphix

Source

FastSource Database

Target HostVirtual

Database

NFS

Fiber

Fiber

RMAN over TCP

No Data Movement

Page 52: Kscope 2013 delphix

Source

Fresh

Virtual Database

Fiber

Page 53: Kscope 2013 delphix

Source

Frequent

Virtual Database

Virtual Database

Target Hosts

Virtual Database

Virtual Database

Fiber

Page 54: Kscope 2013 delphix

Full clones

Page 55: Kscope 2013 delphix

Self Service

Page 56: Kscope 2013 delphix

1 b) Branching and Rapid QA

Page 57: Kscope 2013 delphix

dSource

1 b) Branching

Developer VDB

QA VDB

Page 58: Kscope 2013 delphix

Devv2.6 v2.6v2.6

QA UAT

v2.6

v2.6 v2.6v2.6v2.7

v2.6 v2.6v2.6v2.8

v2.6v2.6 v2.6v2.6

v2.6v2.7 v2.6v2.7

v2.6v2.8 v2.6v2.8

Page 59: Kscope 2013 delphix

Devv2.6 v2.6v2.6

QA UAT

v2.6Production

v2.6 v2.6v2.6v2.7

v2.6 v2.6v2.6v2.8

Source Control for the database data

v2.6v2.6 v2.6v2.6

v2.6v2.7 v2.6v2.7

v2.6v2.8 v2.6v2.8

Page 60: Kscope 2013 delphix

DevProd

2.6

Page 61: Kscope 2013 delphix

Dev

QA

Prod

2.6

Page 62: Kscope 2013 delphix

Dev

QA

UAT

Prod

2.6

Page 63: Kscope 2013 delphix

Dev

QA

UAT

Prod

Dev

QA

UAT

2.6

2.7

Page 64: Kscope 2013 delphix

Dev

QA

UAT

Prod

Dev

QA

UAT

2.6

2.7

Dev

QA

UAT2.8

Page 65: Kscope 2013 delphix

Dev

QA

UAT

Prod

Dev

QA

UAT

2.6

2.7

Dev

QA

UAT2.8

Data Control = Source Control for the Database

Page 66: Kscope 2013 delphix

Dev

QA

UAT

Dev

QA

UAT

2.6

2.7

Dev

QA

UAT

2.8

Data Control = Source Control for the Database

Production Time Flow

Page 67: Kscope 2013 delphix

1 c) Federated Cloning

Page 68: Kscope 2013 delphix
Page 69: Kscope 2013 delphix

Source2

Source3

Source1

1 c) Federated sources

Virtual Database

Virtual Database

Virtual Database

Virtual Database

Page 70: Kscope 2013 delphix

“I looked like a hero”Tony Young, CIO Informatica

Page 71: Kscope 2013 delphix

1. Review Development Use Cases

a) Developer each get a copy

b) QA

c) Federated

Page 72: Kscope 2013 delphix

2. Recovery, Testing, Forensics

a) Forensicsb) A/B testingc) Recovery

Page 73: Kscope 2013 delphix

Source

2 a) Forensic Analysis

Virtual Database

Page 74: Kscope 2013 delphix

Source

2 b) Upgrades, Patches, RAT, A/B

Virtual Database

Page 75: Kscope 2013 delphix

• Production vs Virtual– invisible index on Prod– Creating index on virtual

• Flashback vs Virtual• Keep tests for compare

2 b) Upgrades, Patches, RAT, A/B

Page 76: Kscope 2013 delphix

2 c) Recovery

Page 77: Kscope 2013 delphix

Source

2 c) Logical Recovery Production

Virtual Database

Page 78: Kscope 2013 delphix

Source

2 c) Logical Recovery Development

Virtual Database

VDB rolled back

Page 79: Kscope 2013 delphix

Source

Recovery

VDB

V2P

Page 80: Kscope 2013 delphix

2. Recovery, Testing, Forensics

a) Forensics

b) A/B testing

c) Recovery : Logical and physical

Page 81: Kscope 2013 delphix

3: reportinga) Fast refreshes

b) Temporal queries

c) Confidence testing

Page 82: Kscope 2013 delphix

Fast Refreshes

• Refresh in minutes• Without data movement• Faster , cheaper

Page 83: Kscope 2013 delphix

Temporal Data

Page 84: Kscope 2013 delphix

3: reportinga) Fast refreshes

b) Temporal queries

c) Confidence testing

Page 85: Kscope 2013 delphix

Review: Use Cases

1. Developmenta) Full, Fresh, Fast , Self Serveb) Branchingc) Federated

2. Recovery, Testing :a) Forensicsb) Testing : A/B, upgrade, patchc) Recovery: logical, physical

3. Reportinga) Fast refreshb) Temporal Datac) Confidence testing

Page 86: Kscope 2013 delphix
Page 87: Kscope 2013 delphix

over 10 times

"perhaps the single largest storage consolidation opportunity history“

Page 88: Kscope 2013 delphix
Page 89: Kscope 2013 delphix
Page 90: Kscope 2013 delphix
Page 91: Kscope 2013 delphix

Oracle 12c

Page 92: Kscope 2013 delphix

80MB buffer cache ?

Page 93: Kscope 2013 delphix

200GBCache

Page 94: Kscope 2013 delphix

5000

Tnxs

/ m

inLa

tenc

y

300 ms

1 5 10 20 30 60 100 200

with

1 5 10 20 30 60 100 200Users

Page 95: Kscope 2013 delphix

8000

Tnxs

/ m

inLa

tenc

y

600 ms

1 5 10 20 30 60 100 200Users

1 5 10 20 30 60 100 200

Page 96: Kscope 2013 delphix

Database Virtualization