RecoverPoint Deep Dive

Copyright © 2014 EMC Corporation. All Rights Reserved.

RecoverPoint Deep Dive for PSE

Version 1.0, Feb 2015Kevin Wang

Heterogeneous storage systems data protection solution

RecoverPoint Theory of Operations Introduction


RecoverPoint Overview


RecoverPoint Versions

3 Types

RP Versions RP Zones Splitters RP Volumes

SE (Standard)EX (Extended)CL (Classic)


SAN Environment

SAN(Fabric A)

SAN(Fabric B)

Host

UU

1. Host connected to fabrics

2. Host zoned to the array3. Host masked to

volumes devices on array

4. Host applications writing to volume devices

U – User Volumes (Production)Some Application Running


RecoverPoint Components

RecoverPoint Appliance (RPA)• Manage all aspects of reliable data replication and they can be physical hardware or virtual machines.

Repository Volume• Holds metadata and configuration information

Production Volume• Device that an application is writing to

Journal Volume• Dedicated to holding point-in-time information (also known as snapshots) for each copy/replica

Local Copy Volume• Replica device found on the same Recoverpoint Cluster with the production volume

Remote Copy Volume• Replica device found on a remote Recoverpoint Cluster.

Array-Based Splitter• Types: CX-3, CX-4, VNX, VMAX, VPLEX


RecoverPoint Cluster Environment

Host

Storage

Array

SAN(Fabric A)

SAN(Fabric B)

1. Install RPAs (Min 2 – Max 8)

2. Dual connect RPAs to fabrics

3. Config LAN & WAN ports

4. Zone RPAs into same zone

5. Mask array devices to RPAs

6. Initialize Repository volume

RPA2

RPA1

Repository

UU

U – User Volumes (Production)R – Repository Volume (Metadata)

R

Some Application Running


RecoverPoint Zones & Splitters

3 Types


SE (Standard)EX (Extended)CL (Classic)

Host -> StorageRPA -> StorageRPA -> RPA

VNX/ClariionVMaxVPlex


RecoverPoint (CDP)

Host

Storage

Array

SAN(Fabric A)

SAN(Fabric B)

RPA2

RPA1

R – Repository Volume (Metadata)PV – Production VolumePJ – Production JournalLC – Local Copy (Replica)LJ – Local JournalR

PJ LJPV LC



RecoverPoint (CRR)

SAN SAN SAN SANWAN

PV

PJ

RPA1 RPA1

RPA2RPA2

RC

RJ

RR

R – Repository Volume (Metadata)PV – Production VolumePJ – Production JournalRC – Remote Copy (Replica)RJ – Remote Journal

Production Site DR Site


RecoverPoint (CLR)

SAN SAN SAN SANWAN

PV

PJ

RPA1 RPA1

RPA2RPA2

RC

RJ

RR

R – Repository Volume (Metadata)PV – Production VolumePJ – Production JournalLC – Local Copy (Replica)LJ – Local JournalRC – Remote Copy (Replica)RJ – Remote Journal

LJ

LC


DR SiteProduction Site


RecoverPoint Volumes

3 Types


SEEXCL

Host -> StorageRPA -> StorageRPA -> RPA

VNX/ClariionVMaxVPlex

User JournalRepository


Consistency Group & Replication Sets

PV1

RP Cluster 1

RP – RecoverPointCG – Consistency GroupRS – Replication Set


Consistency Group & Replication Sets (CDP)

PJ

PV1 LC

LJ

CG1

RS1

RP Cluster 1


RPA1


Consistency Group & Replication Sets (CRR)

PJ

PV1 RC

RJ

CG1

RS1

RP Cluster 1 RP Cluster 2


RPA1RPA1


Consistency Group & Replication Sets(CLR)

PJ RJ

CG1RS1


PV1 RC1LC

LJ


RPA1RPA1


Consistency Group & Replication Sets (CLR)


PJ RJ1

CG1RS1

RP Cluster 1 – RPA1 RP Cluster 2 – RPA1

PV1 RC1LC

LJ

RP Cluster 3 – RPA1

RJ2

RC2


Consistency Group (CLR)


CG1

RP Cluster 1 – RPA1 RP Cluster 2 – RPA1RP Cluster 3 – RPA1

PJ RJ1LJ RJ2

RS1

PV1 RC1LC RC2

RS2

PV2 RC1LC RC2

Max RS in CG = 8192 (Physical RPAs)Max RS in CG = 2048 (vRPAs)


CG1

RS1

PJ RJPV1 RC1LCLJ

PJ RJPV2 RC1LCLJ

RS1

CG2

Group Sets


RP – RecoverPoint GS – Group SetCG – Consistency GroupRS – Replication Set

RPA1

RPA2


CG1

RS1

PJ RJPV1 RC1LCLJ

PJ RJPV2 RC1LCLJ

RS1

CG2

Group SetsRP Cluster 1 RP Cluster 2

RP – RecoverPoint GS – Group SetCG – Consistency GroupRS – Replication Set

RPA1

RPA2

GS1Max CGs in GS = 128


Write Phase (CDP, CRR, CLR)

PV PJ LC LJ

Storage Array

SAN

RPA-1

RPA-2

SAN

4

1. The production host writes data to the production volumes which is intercepted by the splitter. 2. The splitter makes a copy of the write and sends the write to the RPA.3. RPA ACK back to the splitter4. The splitter sends write marking metadata to the production journal.5. Splitter send write to the storage system.6. The storage system returns an ACK to the splitter.7. The splitter sends an ACK to the host

5

6

73

1

2

Copyright © 2014 EMC Corporation. All Rights Reserved. 21

Transfer Phase (CDP)

PV PJ LC LJ

Storage Array

SAN

RPA-1

RPA-2

SAN

1. RPA sends write to local journal (not considered a transfer)

1


Distribution Phase (CDP)

PV PJ LC LJ

Storage Array

SAN

RPA-1

RPA-2

SAN

1. Local journal sends write to local copy (default: 5 Phase Distribution is used)1


Transfer (CRR)

PV PJ

Storage Array

SAN

RPA-1

RPA-2

SAN

RC RJ

Storage Array

SAN

RPA-1

RPA-2

SANFC/WAN

1

1. Source RPA sends write to target RPA2. Target RPA ACK back to source RPA3. Target RPA sends write to remote journal

32


Distribution (CRR)

PV PJ

Storage Array

SAN

RPA-1

RPA-2

SAN

RC RJ

Storage Array

SAN

RPA-1

RPA-2

SANFC/WAN

1

1. Remote journal sends write to remote copy (default: 5 Phase Distribution is used)


Write, Transfer, Distribution (CLR)

PV PJ LC LJ

Storage Array

SAN

RPA-1

RPA-2

SAN

RC RJ

Storage Array

SAN

RPA-1

RPA-2

SANFC/WAN


Write (CLR)

PV PJ LC LJ

Storage Array

SAN

RPA-1

RPA-2

SAN

RC RJ

Storage Array

SAN

RPA-1

RPA-2

SANFC/WAN

45

6

73

1

2

1. The production host writes data to the production volumes which is intercepted by the splitter. 2. The splitter makes a copy of the write and sends the write to the RPA.3. RPA ACK back to the splitter4. The splitter sends write marking metadata (Delta Marking) to the production journal.5. Splitter send write to the storage system.6. The storage system returns an ACK to the splitter.7. The splitter sends an ACK to the host


Transfer (CLR)

PV PJ LC LJ

Storage Array

SAN

RPA-1

RPA-2

SAN

RC RJ

Storage Array

SAN

RPA-1

RPA-2

SANFC/WAN

1. Source RPA sends write to target RPA2. Target RPA ACK back to source RPA (* Note: Delta Marking data released in production journal)3. Target RPA sends write to remote journal3. Source RPA sends write to local journal

12

33

*


Distribution (CLR)

PV PJ

Storage Array

SAN

RPA-1

RPA-2

SAN

RC RJ

Storage Array

SAN

RPA-1

RPA-2

SANFC/WAN

1

1. Remote journal sends write to remote copy (default: 5 Phase Distribution is used)1. Local journal sends write to Local copy (default: 5 Phase Distribution is used)

1


RecoverPoint – Journal Volumes


RecoverPoint – Journal Volume

Snapshots75%

Snapshots75%Snapshots75%



Snapshots75%

Snapshots75%

Target Side Logging (TSL)Image Access Logging (IAL)

20%

Snapshots75%

Target Side Logging (TSL)20%



Snapshots75%

Snapshots75%


20%

Snapshots75%


SystemMarking Mode

Pointers5%

SizeFor Normal CG:Min: 10GB

For Distributed CG:Min: 40GB

Max Size:10TB

CG = Consistence Group


Journal Volume Snapshot Space

Undistributed writes (DO stream)

Distributed writes (UNDO stream)

Replica(Copy)

Source/TargetRPAFrom

Splitter orSource RPA

Oldest Writes

Oldest Writes

75% sectionof Journal





Replica

Source or TargetRPA

Oldest Writes

Oldest Writes

From Splitter orSource RPA

NewWrites

NewWrites

New writes are writtento the DO stream ofthe journal






Replica

Source or TargetRPA

Oldest Write

Oldest Write


Write

Blockto

Change

Write will overwriteblock on copy






Replica

Source or TargetRPA

Oldest Write

Oldest Write


Write

Blockto

Change

Before write from the DO stream is applied to the copy, the old data in that block is written to the UNDO stream

OldData






Replica

Source or TargetRPA

Oldest Write

Oldest Write


The write from the DO streamis than applied to the copy

OldData

Write



5 Phase Distribution

DO

UNDO

JV

RemoteReplica

Target RPA

Write(from source)

1

From SourceRPA

Oldest

Oldest



DOUNDO

JV

Replica

Target RPA

Write(from source)

1

2Read Oldest Write from DoStreamnew

From SourceRPA

Oldest

Oldest



DO

UNDO

JV

Replica

Target RPA

Write(from source)

1

2

new

Read

old

3

Read Oldest Write from DoStream

From SourceRPA

Oldest

Oldest



DO

UNDO

JV

Replica

Target RPA

Write(from source)

1

2 Read

new

Read

old

3

Write4

old

From SourceRPA

Oldest

Oldest



DO

UNDO

JV

Replica

Target RPA

Write(from source)

1

2 Read

new

Read3

Write4

old

Write

new

5

From SourceRPA

Oldest

Oldest


3 Phase Distribution ExampleWhat triggers going into 3 Phase Distribution:1. When the write-rate of the production host is greater than the distribution-rate of five-phase distribution (This, over time, can cause the queue of snapshots waiting for distribution to reach the maximum journal capacity).2. When the journal lag exceeds the Maximum Journal Lag setting.


3 Phase Distribution (Fast Forwarding)

DO

UNDO

JV

Replica

Target RPA

Write(from source)

1

1 Discards all of the snapshots (Undo – Already distributed)

From SourceRPA

Oldest



DOUNDO

JV

Replica

Target RPA

Write(from source)

1

2 Read

new

1 Discards all of the snapshots (Undo – Already distributed)

From SourceRPA

Oldest



DO

UNDO

JV

Replica

Target RPA

Write(from source)

1

2 Read

new

Write

new

3From SourceRPA

Oldest


1 Phase Distribution (Fast First Time Initialization)

DO

UNDO

JV

Replica

Target RPAWrite(from source)

1

new

Journal volumeIs bypassed

From SourceRPA


Journal Sizing


• You need to know: Write change rate of the

application in seconds Required rollback time in

seconds

Journal Size Requirements

Snapshots75%

Snapshots75%


20%

Snapshots75%


SystemMarking Mode

Pointers5%


Journal Size = 1.05 x [(Δ data per second) x (required rollback time in seconds) ÷

(1 - target side log size)] + reserved for marking

Journal Size Formula

Snapshots75%

Snapshots75%


20%

Snapshots75%


SystemMarking Mode

Pointers5%


Journal Size = 1.05 x [( ) x ( ) ÷



Snapshots75%

Snapshots75%


20%

Snapshots75%


SystemMarking Mode

Pointers5%

Δ data per second

Required rollback time in seconds


Journal Size = x [(Δ data per second) x (required rollback time in seconds) ÷



Snapshots75%

Snapshots75%


20%

Snapshots75%


SystemMarking Mode

Pointers5%

1.05



( )] + reserved for marking


Snapshots75%

Snapshots75%


20%

Snapshots75%


SystemMarking Mode

Pointers5%

1 - target side log size



(1 - target side log size)] +


Snapshots75%

Snapshots75%


20%

Snapshots75%


SystemMarking Mode

Pointers5%

reserved for marking

• Add 1.5GB for Normal Consistency Group• Add 6GB for a Distributed Consistency Group




• Δ data per second = 5Mbps• Required rollback time in seconds = 86400• Target side logging = 20%• Reserved for marking = 1.5GB

Normal Consistency Group

Journal Sizing Example


Journal Size = 1.05 x [(5Mbps) x (86400) ÷ (1 - .20)]

+ 1.5GB

• Δ data per second = 5Mbps• Required rollback time in seconds = 86400

24 hours

• Target side logging = 20%• Reserved for marking = 1.5GB




Journal Size = 1.05 x [(432000) ÷ (.80)]

+ 1.5GB


24 hours





Journal Size = 1.05 x 540000+ 1.5GB


24 hours





Journal Size = 567000Mb + 1.5GB


24 hours





Journal Size = (567000Mb) / (8 * 1024)+ 1.5GB


24 hours




NOTE: 567000Mb needs to be convertedto GB


Journal Size = 69.21GB + 1.5GB • Δ data per second = 5Mbps• Required rollback time in seconds = 86400

24 hours





Journal Size = 70.71GB• Δ data per second = 5Mbps• Required rollback time in seconds = 86400

24 hours





Journal Devices

Snapshots75%

Snapshots75%


20%

Snapshots75%


SystemMarking Mode

Pointers5%

40GB30GB

10GB

If the devices that make up thejournal are different sizes andnot within 85% of each other, the journalwill be a concatenation of the devices

LUNLUNLUN


Journal Devices

Snapshots75%

Snapshots75%


20%

Snapshots75%


SystemMarking Mode

Pointers5%

20GB20GB20GB20GB

If the devices that makes up thejournal are the same size orwithin 85% of each other, writeswill be stripped across all thedevices.

LUNLUNLUNLUN


Journal Devices

Snapshots75%

Snapshots75%


20%

Snapshots75%


SystemMarking Mode

Pointers5%

20GB20GB20GB20GB

Additional devices can be added to thejournal on the fly. However, in this exampleyou will have a concatenation of the twostripped device sets.

LUNLUNLUNLUN

20GB 20GB

LUN LUN


Journal Devices

Snapshots75%

Snapshots75%


20%

Snapshots75%


SystemMarking Mode

Pointers5%

20GB20GB20GB

If you want the journal to be a six devicestripe, you can do one of the following:1. Remove one of the devices and replace it2. Disable the consistency group and than re-enable it

LUNLUNLUN

20GB 20GB

LUN LUN


Journal Devices

Snapshots75%

Snapshots75%


20%

Snapshots75%


SystemMarking Mode

Pointers5%

If you want the journal to be a six devicestripe, you can do one of the following:1. Remove one of the devices2. Disable the consistency group

20GB

LUN

20GB 20GB 20GB 20GB 20GB

LUN LUN LUN LUN LUN

The journal will be rebuilt as a six devicestripe, but all previous snapshots willbe lost.


Image Access


Replica

Snapshots75%

Snapshots75%

20%

Snapshots 75%

TSL 20%

System 5%

From SourceRPA

Image Access – Logged (Physical)

75% region

Do

Undo

1. Test a copy, failover,or recover productionoperation started.2. PIT (Snapshot)selected.

Newest

Newest

Newest

TransferNew WritesTo DO stream


Replica

Snapshots75%

Snapshots75%

20%

Snapshots 75%

TSL 20%

System 5%

From SourceRPA


75% region

Do

Undo

123

1. Roll PITsforward to replica (copy)

1 2 3

Newest

Newest


Roll forwardthe copy


Replica

Snapshots75%

Snapshots75%

20%

Snapshots 75%

TSL 20%

System 5%

From SourceRPA


75% region

Do

Undo

21

1. Roll PITsback to replica (copy)

1 2 3

Newest

Newest


3Roll backthe copy


Replica

Snapshots75%

Snapshots75%

20%

Snapshots 75%

TSL 20%

System 5%

From SourceRPA


75% region

Do

Undo

1. Copy readyfor access2. Host mountsthe copy3. Transfers todo stream occurring

20% - TSL



Replica

Snapshots75%

Snapshots75%

20%

Snapshots 75%

TSL 20%

System 5%

From SourceRPA


75% region

Do

Undo

Host makes writes to the copy. Old data written to TSL region in journal

20% - TSL

old

old

old

old

oldold



Replica

Snapshots75%

Snapshots75%

20%

Snapshots 75%

TSL 20%

System 5%

From SourceRPA


75% region

Do

Undo

New data writtento copy

20% - TSL

new

old

new

new

oldold



Replica

Snapshots75%

Snapshots75%

20%

Snapshots 75%

TSL 20%

System 5%

From SourceRPA


75% region

Do

Undo

1. Logged access ended.Old data from TSL regionwritten back to copy.2. Host unmounts copy.3. 5-Phase Distribution restarts

20% - TSL

oldoldold



Replica

Snapshots75%

Snapshots75%

20%

Snapshots 75%

TSL 20%

System 5%

From SourceRPA

Image Access - Virtual

75% region

DoUndoNewest at topOldest at Bottom



Replica

Snapshots75%

Snapshots75%

20%

Snapshots 75%

TSL 20%

System 5%

From SourceRPA

Image Access - Virtual

75% region

DoUndo

VV



Replica

Snapshots75%

Snapshots75%

20%

Snapshots 75%

TSL 20%

System 5%

From SourceRPA

Image Access - Direct

75% region

DoUndo

Host is given permanentaccess to the copy


Replica

Snapshots75%

Snapshots75%

20%

Snapshots 75%

TSL 20%

System 5%

From SourceRPA

Image Access - Direct

75% region

1. Transfers is stopped2. Journal is deleted


Recoverpoint Debug


Part1 – General Config Check



8F,SANC,SHOW,SESS



• When a consistency group is enabled, an Open Replicator RecoverPoint session is created for each production volume within that consistency group.

• 8F,SANC,SHOW,SESS,,’device’ will give more information on a particular RP session.



We can use symaccess commands to check storage groups etc – note the naming structure will usually have ‘RPA’ for the RP storage provisioning.



• We can see 8 gatekeeper devices assigned to each RP – this is as per best practice.

• RPA1 has devices 0D1 – 0D8.



Note the gatekeeper devices are small – 3MB.



The ‘cluster’ masking view will show us it includes the Source, Journal and Repository storage groups



• 00C7 is the repository device. This will show as a ‘target’ of a recoverpoint session – ACFS.

• 026D – 27C are the production journal devices. This will show as a ‘target’ of a recoverpoint session – ACFS.

• Metas 111, 113, 116, 14D, 152 are the source devices - These will show as a part of a recoverpoint (SANCopy) session – SANC.



• On the remote side you will see sessions if you enable image access. • Open Replicator RecoverPoint sessions are not created for the copy volumes until

image access is enabled. Even after image access is disabled, stopped Open Replicator RecoverPoint sessions remain for each copy volume.

• The session is now showing as ‘stopped’ as image access has since been disabled. (This session will be removed if you remove the Consistency Groups)



• We can again check the masking views to decipher how the storage volumes are assigned.



• 008B is the repository device. This will show as a ‘target’ of a recoverpoint session – ACFS.

• 0053 – 0062 are the copy journal devices. This will show as a ‘target’ of a recoverpoint session – ACFS.

• Metas 73, 75, 199, 1D0, 207 are the replica devices - These will show as both Source (when image access was enabled) and Target (for the CRR recoverpoint session) – CASC.


Part2 – RP Debug Example

• 0124.0E errors (Reached count of host commands aborted because of SANCopy problems)

• See the following example of 0124s occurring against device 1039 on FA5E0.

• This error indicates a timeout after the IO was split out to the RPAs.



• Checking the E2,,QREC we see the DID (Destination ID) for this IO was CF400.• The CDB of E9 indicates it occurred an ORS donor update.• Donor update is used for the default mode of splitting IO to the RPA. When a host write come

s in, we use a donor update command to push a copy of it to the RPA. So for RP sessions, during normal operations, almost all the commands you will see are donor updates.

• The ORS Push (CDB of E5) would be used during an initial RP Synchronization.

Also note the EB output points to port 0 but the E2,,QREC points to port 1 – Why?

This is because the write came in on port 0 but the splitter sent the IO out port 1. (Normal operation is that the host IO is split out the same FA port)



We can confirm what port the device is mapped to using the A7, FMAP – We see it’s mapped to port 0 on FA5E. This means the host writes come in on this port.

The A7 command will show it is in an Open Replicator RP session. RP Device tagging informs the Symmetrix array that the device is reserved for RecoverPoint replication.



• A7, FMTA will show us the meta is 1038.

• Using 8F,DVIN,VIEW,1 we see FCID CF400 logging in on port 1. • We also see the WWN corresponding to FCID CF400 is 5001248231387507. • The ‘5001248’ of the WWN indicates an RPA.



• So at this point we can say we have write going to device 1039 on FA5E0 being split through FA5E1 to the RP. Once status is returned to by the RP to the FA, the FA can return status to the host.

• However if the RP takes more than 11 secs to respond the FA will fail the session with 0124.0E, and abort the IO.

• This is not a fatal error, so the host should retry the write.• In this case the 0124s were streaming against the same FA/RPA so the RP logs had to be

reviewed to investigate the delays from the RPA ACK'ing to the FA.


Q & A


THANK YOU

RecoverPoint Deep Dive

Documents

Transcript of RecoverPoint Deep Dive