Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader...

43
Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader [email protected]

Transcript of Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader...

Page 1: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

Host Based Volume Shadowing Update

Jeff FriedrichsVolume Shadowing Project Leader

[email protected]

Page 2: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

May 2004

Agenda

• Review of Recent Features• New Features to look forward to

Page 3: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

May 2004

HBMM

Host Based Mini Merge

Available on V7.3-2 (with TIMA kits)

Page 4: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

May 2004

What is Mini Merge?

• Full merge requires comparing entire shadow set

• But only blocks with I/O in progress need to be merged

• MSCP Based mini merge – Supported on HSCxx / HSDxx / HSJxx controllers– Controller tracks in-progress writes– Host can get list of writes from the controller

• Host Based Mini Merge (HBMM)– Selected cluster hosts track recent writes using write

bitmap– Bitmap is periodically reset to flush out old writes– Contents of bitmap drive mini merge operation

Page 5: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

May 2004

Write Bitmap for Mini Merge

• Write bitmap originally released on V7.2-2 for mini copy– Each system has an in-memory bitmap to track write I/O– Each bitmap has one system that is the master– 2KB memory per Gbyte of storage per bitmap per system

• Originally 6 bitmaps, HBMM extended to 12– There are 6 bitmaps are available (per VU) for mini merge use– There are also 6 bitmap slots reserved for mini copy use

• HBMM recovery must be by a bitmap master system

• Use multiple bitmap masters for availability of the bitmap after a system crash

Page 6: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

May 2004

MINI COPY Bitmaps

• Created with $DISMOUNT/POLICY=MINICOPY

• NO redundancy

• Uses bitmap first 6 bitmap slots

Page 7: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

May 2004

HBMM Bitmap Master Policy

• The policy defines– number of bitmap masters for a VU in the cluster– the location of masters (in counted groups)– the bitmap reset threshold

• Named policies– Are known cluster wide– Any named policy can be assigned to an individual

VU or to multiple VUs– A named policy can be deleted at any time

• That does not affect VUs that had it applied

Page 8: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

May 2004

Bitmap Master Policy

• A policy must be directly assigned to VU for HBMM to be enabled on that VU

• With the following exception– If a DEFAULT policy has been defined, then every VU in

the cluster, that does not have a named policy, will automatically “pick up” the DEFAULT policy

– To prevent a VU from “picking up” the DEFAULT policy use • SET SHADOW DSA1: / POLICY = HBMM = NODEFAULT

• In summary– the DEFAULT policy and a mix of VU specific policies

can be used in the cluster at the same time

Page 9: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

May 2004

Policy Definition

SET SHADOW /POLICY = HBMM = (

(MASTER_LIST = (NODE1,NODE2,NODE3), COUNT=2),

(MASTER_LIST = (NODE4,NODE5,NODE6), COUNT=2),

RESET_THRESHOLD=50000 ) / NAME = POLICY_1

SET SHADOW DSA1: / POLICY = HBMM = POLICY_1

Page 10: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

May 2004

Policy Definition continued

SET SHADOW DSA1: /POLICY = HBMM = (

(MASTER_LIST = (NODE1,NODE2,NODE3), COUNT=2),

(MASTER_LIST = (NODE4,NODE5,NODE6), COUNT=2),

RESET_THRESHOLD=50000 )

Page 11: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

May 2004

Other HBMM Controls

SET SHADOW DSA1: / DISABLE = HBMM– Disables HBMM on DSA1

SET SHADOW DSA1: / ENABLE = HBMM– Enables HBMM on DSA1– If there is a policy for the VU to enable

SET SHADOW DSA1: / POLICY=HBMM=(MASTER=*)– Allows all systems in the cluster to be bitmap masters – First six to mount DSA1: will become masters

– Others eligible when a master dismounts VU or crashes

Page 12: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

May 2004

Other HBMM Controls

SET SHADOW / POLICY = HBMM = (MASTER_LIST = …. )/ NAME = DEFAULT

– Creates default policy for all shadow sets that are mounted in the cluster that do not have a VU specific policy in place

SET SHADOW DSA1: / POLICY = HBMM = NODEFAULT– Disables HBMM for DSA1– Can be used to override a DEFAULT policy on a specific VU– Can be viewed as a policy governor

SET SHADOW / POLICY = HBMM / NAME=POLICY_1 / DELETE– Will delete the policy named “POLICY_1”

Page 13: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

May 2004

HBMM rules

– If a policy is associated with a VU, HBMM is automatically enabled upon first mount on a system that has been named as bitmap master in that policy

– If a master system ceases to be a master (due to dismount or crash), a new master bitmap will be automatically started on another master system, subject to the policy in force on the VU

– Devices capable of MSCP Based (HSC/HSJ/HSD) mini merge are not eligible for HBMM

– To enable host based mini merge on a VU, all systems that mount the VU must be HBMM capable

Page 14: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

May 2004

AMCVP

• Automatic Mini Copy on Volume Processing• aka Multiuse Bitmaps• Available in V8.3• Extension of HBMM

Page 15: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

May 2004

AMCVP Considerations

• Multiuse bitmaps are valid for both mini MERGE and mini COPY operations

• Multiuse bitmaps are NOT zeroed, therefore mini MERGEs will take longer and longer

• Multiuse bitmaps must be deleted manually if the removed member is never coming back to the set

Page 16: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

May 2004

Policy Definition

SET SHADOW /POLICY = HBMM = (

(MASTER_LIST = (NODE1,NODE2,NODE3), COUNT=2, MULTIUSE=2),

(MASTER_LIST = (NODE4,NODE5,NODE6), COUNT=2, MULTIUSE=1),

RESET_THRESHOLD=50000 ) / NAME = POLICY_1

SET SHADOW DSA1: / POLICY = HBMM = POLICY_1

Page 17: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

May 2004

Write Bitmap Message Modes

• Single Message Mode– Only one logical WBM write request per SCS

message

• Buffered Message Mode– Supports up to nine logical WBM write requests per

SCS message– Message dispatching occurs

• when the message is full

• when an internal WBM timer expires

Page 18: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

May 2004

Write Bitmap Message Mode Control

SYSGEN Parameters for WBM

• WBM_MSG_UPPER– The upper threshold for the number of messages

sent during the calibration interval that will initiate buffered message mode

• WBM_MSG_LOWER– The lower threshold for the number of messages

sent during the calibration interval that will initiate single message mode

V8.3 and V8.3-1H1 – CHECK YOUR PARAMETERS!

Page 19: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

May 2004

Merge and Copy Prioritization

Page 20: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

May 2004

Current Merge and Copy Controls

• Management controls to determine order and choice of system for copy and merge operations are incomplete.

• Important volumes may be merged after less important volumes

• Systems better suited to perform merge or copy operations on some volumes are not always selected

Page 21: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

May 2004

Merge and Copy Control

• Allow user to assign a priority to every VU

• Better predict which system will perform any transient state operations (merge or copy operations) – requires SYSGEN settings

• Utilize SHADOW_MAX_COPY dynamic characteristic

Page 22: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

May 2004

Shadow Priority

• New command qualifier– $ SET SHADOW /PRIORITY = n DSAnnnn:

• A range of 0 through 10,000

• Default is 5000

• 1 is the lowest priority

• Zero has special meaning

• At MOUNT time each VU will be placed in system wide priority linked list by this value– VUs at the same priority have an undefined ordering

• Governs merge and copy priority for VUs on this system

Page 23: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

May 2004

Show Shadow Priority

New command $ SHOW SHADOW /BY_PRIORITY

• Lists the DSA devices on this system using the priority assigned to each, highest to lowest

• Shows transient state % and system performing operation

Device Priority Virtual Unit State % Completed on Node

DSA3233: 3233 Steady State DSA2325: 2325 Not Mounted DSA42: 42 Full Merge Active 14% on CSGF1

Page 24: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

May 2004

SYSGEN parameter

SHADOW_REC_DLY (Shadow Recovery Delay)

• This parameter governs how many seconds, after the VU enters a merge transient state, that this system will wait before it attempts to manage that state

• Default of 20 seconds

• Making this value different across the cluster will guide which system will manage transient state operations

Page 25: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

May 2004

Using Priority List

Until all SHADOW_MAX_COPY threads are used on a system, the priority list is processed in two phases:

– Host Based Mini Merge or MSCP Based Mini MergeThen

• Full Copy state

or • Full Merge state

This implies that all HBMM VUs are processed before any full copy or full merge VUs … regardless of the priority value for any VU

Page 26: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

May 2004

Per SSM Served Delay

Each SSM on an MSCP served path has an additional delay added before attempting to obtain the _COPIER lock to manage a merge operation.

If the default value (30 seconds) is not suitable it can be changed

• SET SHADOW/<tbs>=n DSAn:– Will use this value for all SSMs on an MSCP served

path• SET SHADOW/<tbs>=n /ALL

– Will change the default value of 30 that HBVS will use for all VUs that are mounted on this system

– Will change the value on all VUs currently mounted on this system

Page 27: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

May 2004

Managing Transient State Operations

New SET SHADOW command qualifiers:/EVALUATE=RESOURCES

– Is a system specific command … i.e. not cluster specific • Thus it only affects the VUs that are being merged or

copied by this system

– Useful if the command is issued before• SHADOW_MAX_COPY is increased or is decreased

– To start or stop a merge or a copy operation

• The priority of a VU is changed– Priority of 0 now non-zero or the inverse– Priority value has been raised or lowered

Page 28: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

May 2004

Managing Transient State Operations

New SET SHADOW command qualifier:/DEMAND_MERGE DSAn:Changes the state of the VU to a merge required state

• The type of merge initiated will depend on merge recovery characteristics that are enabled currently on the VU

• To insure that a full merge is initiated on a VU• Disable host based mini merge that it is currently enabled

• This is especially useful if the shadow set had been created with INIT/ SHADOW without / ERASE

• Can be used if differences are found on the members with ANALYZE /DISK /SHADOW

Page 29: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

May 2004

The Future of Shadowing

V8.4 Enhancements

Page 30: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

May 2004

HBMM Dismount

• Further extension of HBMM and AMCVP

• Allows all 12 bitmap slots to be used by HBMM or MiniCopy

• Provides bitmap redundancy for removed members

Page 31: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

May 2004

Policy Definition

SET SHADOW /POLICY = HBMM = (

(MASTER_LIST = (NODE1,NODE2,NODE3), COUNT=2, MULTIUSE=2, DISMOUNT=2),

(MASTER_LIST = (NODE4,NODE5,NODE6), COUNT=3, MULTIUSE=2, DISMOUNT=2),

RESET_THRESHOLD=50000 ) / NAME = POLICY_1

Page 32: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

May 2004

Fast MINI operations

Performance enhancement for Mini Copy and Mini Merge

Look ahead in the bitmap

No user intervention required

Significantly reduces I/O from SHADOW_SERVER

Page 33: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

May 2004

Write Bitmap Enhancements

• Fix buffered mode transitions

• Multicast “set bit” messages

• Optimize sequential writes

Page 34: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

May 2004

SPLIT_READ_LBNS

• Attempt to utilize read ahead controller cache

• Currently reads are “round robin” among members

• Simply divide the virtual unit into smaller chunks based on LBN and send all reads for the same area to the same member

• Only members with the lowest and same READ_COST are used

Page 35: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

May 2004

SPLIT_READ_LBNS

$ show shad dsa33/full

Device $1$DGA33 Index 0 Status 000000A0 mbr_src,mbr_valid Ext. Member Status 00 Read Cost 2 Site 0 Member Timeout 120 UCB 81DCA540 Error Count 0 LBN_LIMIT: 00000000

Device $1$DGA41 Master Member Index 1 Status 000000A0 mbr_src,mbr_valid Ext. Member Status 00 Read Cost 2 Site 0 Member Timeout 120 UCB 81DB8B80 Error Count 0 LBN_LIMIT: 00000000

Device $1$DGA34 Index 2 Status 200000A0 mbr_src,mbr_valid,user_supplied_read_bia Ext. Member Status 00 Read Cost 3 Site 0 Member Timeout 120 UCB 81DD5E40 Error Count 0 LBN_LIMIT: 00000000

Page 36: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

May 2004

SPLIT_READ_LBNS

$ SET SHADOW/ENABLE=SPLIT_READ_LBNS$ SHOW SHADOW/FULL DSA33

Device $1$DGA33 Index 0 Status 000000A0 mbr_src,mbr_valid Ext. Member Status 00 Read Cost 2 Site 0 Member Timeout 120 UCB 81DCA540 Error Count 0 LBN_LIMIT: 000F4240

Device $1$DGA41 Master Member Index 1 Status 000000A0 mbr_src,mbr_valid Ext. Member Status 00 Read Cost 2 Site 0 Member Timeout 120 UCB 81DB8B80 Error Count 0 LBN_LIMIT: 001E8480

Device $1$DGA34 Index 2 Status 200000A0 mbr_src,mbr_valid,user_supplied_read_bia Ext. Member Status 00 Read Cost 3 Site 0 Member Timeout 120 UCB 81DD5E40 Error Count 0 LBN_LIMIT: 00000000

Page 37: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

May 2004

SPLIT_READ_LBNS

$ SET SHADOW –_$ /DISABLE=SPLIT_READ_LBN DSA1:

Page 38: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

May 2004

One More!

Page 39: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

May 2004

XMBRS

$ show dev DSA6

Device Device Error Volume Free Trans Mnt Name Status Count Label Blocks Count CntDSA16: Mounted 0 XMBRS 49954 1 1$1$DGA1: (CSGF2) ShadowSetMember 0 (member of DSA6:)$1$DGA2: (CSGF2) ShadowSetMember 0 (member of DSA6:)$1$DGA3: (CSGF2) ShadowSetMember 0 (member of DSA6:)$1$DGA4: (CSGF2) ShadowSetMember 0 (member of DSA6:)$1$DGA5: (CSGF2) ShadowSetMember 0 (member of DSA6:)$1$DGA6: (CSGF2) ShadowSetMember 0 (member of DSA6:)

Page 40: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

May 2004

XMBRS

• To turn it on, just specify more than 3 members for the shadow set

$ MOUNT/SHADOW DSA16 –_$ /SHADOW=($1$dga1, $1$dga2, $1$dga3,

$1$dga4) –_$ DSA6

• No new qualifiers• No changes to DISMOUNT

Page 41: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

May 2004

XMBRS Compatibility

• Mixed version, 3 member shadow sets will continue to work

• To use 4 or more members, all systems that have the VU MOUNTed, must have the new software

• MOUNT/INCLUDE of a once XMBR virtual unit on an older version may not find all of the members

Page 42: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

May 2004

XMBRS Performance

• Performance testing in progress• Reads could be faster• Writes *WILL* be slower

• Important to use SET SHADOW commands– Noticeable difference with /COPY_SOURCE

Page 43: Host Based Volume Shadowing Update Jeff Friedrichs Volume Shadowing Project Leader Jeff.Friedrichs@hp.com.

May 2004

Questions??