IBM FlashCopy backup & copy solution for SAP …...HANA SnapShot, see SAP notes 2096000, 2039883....

© Copyright IBM Corporation 2020

IBM FlashCopy backup & copy

solution for

SAP HANA TDI

(Intel / IBM POWER) ESCC, European Storage Competence Center, IBM Germany

Location of this document:

http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP102476

email: [email protected]

Download: https://public.dhe.ibm.com/storage/FCtool/

Forum: https://ibm.biz/BdqBUP

community.ibm.com → Milldleware User Community →

Global Group Storage Management

Version V4.3.1 – March 2020

http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP102476

mailto:[email protected]

https://public.dhe.ibm.com/storage/FCtool/

https://ibm.biz/BdqBUP

IBM FlashCopy backup solution for SAP HANA TDI

Page - 2 © Copyright IBM Corporation 2020

Content

EXECUTIVE SUMMARY ............................................................................................................................ 3

AUDIENCE ............................................................................................................................................... 3

BACKUP CONCEPT FOR SAP HANA .......................................................................................................... 4

CONCEPT OF FCTOOL .............................................................................................................................. 4

SUPPORT MATRIX & REQUIREMENTS ..................................................................................................... 6

SUPPORTED STORAGE SYSTEMS ........................................................................................................................ 6

SUPPORTED HANA CONFIGURATION ................................................................................................................ 7

LIMITATIONS BY SAP ..................................................................................................................................... 7

VMWARE .................................................................................................................................................... 7

LINUX LVM ................................................................................................................................................ 7

INSTALLATION OF IBM SPECTRUM PROTECT FOR ERP .......................................................................................... 8

SSH COMMUNICATION .................................................................................................................................... 8

FCTOOL DEPLOYMENT .................................................................................................................................... 9

HANA RAPID COLD START -- SW RAID ............................................................................................................. 9

SUPPORT FOR SPECTRUM VIRTUALIZE HYPERSWAP ....................................................................................... 10

IMPORTANT: VD REQUIREMENTS ........................................................................................................... 10

INSTALLATION OF FCTOOL .................................................................................................................... 11

FIRST TIME INSTALLATION .............................................................................................................................. 11

MANUAL INSTALLATION OF FCTOOL: ............................................................................................................... 12

UPDATE OF FCTOOL ..................................................................................................................................... 13

ADDING HANA SYSTEMS TO FCTOOL .................................................................................................... 13

PREPARATION ON HANA VM ....................................................................................................................... 13

SETUP HDBUSERSTORE.................................................................................................................................. 14

CONFIGURING FCTOOL ......................................................................................................................... 15

SETUP OF FCTOOL.INI .................................................................................................................................... 15

MULTIPLE FCTOOL.INI FILES ............................................................................................................................ 19

TESTING FCTOOL ......................................................................................................................................... 19

USING FCTOOL ...................................................................................................................................... 22

BACKUP ..................................................................................................................................................... 23

RESTORE .................................................................................................................................................... 24

BACKUP AND RESTORE OF MDC .................................................................................................................... 25

REMOTE .................................................................................................................................................... 25

QUERY ...................................................................................................................................................... 26

AUDIT STORAGE .......................................................................................................................................... 27

REFRESH .................................................................................................................................................... 29

COPY ........................................................................................................................................................ 31

HOW TO DELETE FLASHCOPY BACKUPS ............................................................................................................ 32

HOW FCTOOL USES FLASHCOPY MAPPINGS OF SPECTRUM VIRTUALIZE ............................................... 32

FCTOOL BEST PRACTICES / SCHEDULING ............................................................................................... 33



RECOVER THE SAP HANA SYSTEM ......................................................................................................... 34

RESOURCES ........................................................................................................................................... 45

SERVICE ................................................................................................................................................ 45

SUPPORT & HELP .................................................................................................................................. 45

ERROR MSG .......................................................................................................................................... 45

FCTOOL MESSAGE CODES ..................................................................................................................... 46

HISTORY ............................................................................................................................................... 46

NOTICES ................................................................................................................................................ 47

TRADEMARKS ....................................................................................................................................... 49

Executive Summary

SAP®, and with-it SAP HANA™ is used by almost every industry, and the business-crit-

ical application. Therefore, Business Continuity for SAP is absolute required.

IBM Spectrum Protect for ERP offers on-line data base backup for SAP HANA, ensuring

data is stored securely and efficient.

Depending on the HANA data base size the time needed for on-line backup or on-line

restore varies between minutes to hours.

Here IBM FlashCopy technology can help to reduce this time significantly, down to

some seconds or a few minutes.

The FCtool uses IBM FlashCopy technology, and is capable to backup, restore, and

copy SAP HANA systems.

The following chapters describes the requirements, installation & setup, and how to

use this FCtool.

Audience

This paper is intended as an architecture and configuration guide to setup IBM Spec-

trum Virtualize (SVC or Storwize) and IBM FCtool for SAP HANA TDI within a SAN en-

vironment.

This document has been written for IT technical specialists and architects, with ad-

vanced skill levels on SUSE® Linux™ Enterprise Server, and IBM Spectrum Virtualize

(IBM SAN Volume Controller, SVC) or IBM Storwize™.



Backup concept for SAP HANA

IBM Spectrum Protect for Enterprise Resource Planning (TSM for ERP) protects your

SAP system data. It provides automated data protection for SAP and SAP HANA envi-

ronments, fully integrated with SAP Business Suite and SAP HANA STUDIO.

To reduce the time for the SAP backup and restore, the online backup can be combined

with storage-based backup via IBM FlashCopy. This can be achieved by using the CLI

(command line interface) of IBM Spectrum Virtualize (SVC or Storwize) – from here on

referred as SVC.

The combination of both, online backup and FlashCopy, provides a high secure and

fast way to protect SAP data bases, including SAP HANA.

The IBM FCtool automates the entire FlashCopy process.

Concept of FCtool

FCtool enable you to backup and restore your HANA instance fully automated, by using

IBM FlashCopy of IBM Spectrum Virtualize. Typically, the backup or restore process

by FCtool take less than one minute. The backup via FlashCopy does not replace the

online backup, it is an add-on to accelerate the processes. The FCtool needs to run on

a dedicated VM (or LPAR) and controls IBM Spectrum Virtualize and SAP HANA via

LINUX ssh command.

The FlashCopy backup works on the HANA data base / system level, which means the

entire data based is backed up or restored, with all tenants.



In addition to the backup and restore functions FCtool provides two other com-

mands: copy and refresh, as the following graphic illustrates:

The copy function copies DATA and LOG of the source HANA system to an empty tar-

get system -- the term "HANA system" means the entire data base, with all tenants.

The function refresh replaces DATA and LOG of a target system, similar to redirected

restore. The required SAP post processing is not part of the FCTool refresh function.



Support Matrix & Requirements

This table shows the supported configuration, and the required components:

Components FCtool HANA instances

Dedicate OS (own VM/LPAR) X1 yes

OS: LINUX SLES, RH yes yes

VMware yes physical RDM only

Server: IBM POWER LE only LE only

Server: Intel yes yes

VersaStack yes yes

NPIV -- X

HANA TDI setup -- X

HANA version -- min 2.0 SPS4

IBM SP for ERP 2 -- X

App LAN access X X

Storage LAN access X --

SAN access -- X

IBM Storage Systems -- IBM Spectrum Virtualize >= V8

Legend: X: required, --: not applicable, LE: Little Endian

(1): The LPAR / system needs min 1 GB RAM and 0.2 CPUs (two shared cores).

(2) FCtool does not rely on IBM Spectrum Protect, it is just important that an SAP certified backup

SW is used for HANA, through SAP API BACKINT.

(3) Functions backup and restore only

Supported storage systems

This FCtool supports IBM Storage Systems based on

IBM Spectrum Virtualize:

• Spectrum Virtualize minimum version: 8.0.0

• IBM SVC

• IBM Storwize V7000, V5000

• IBM FlashSystem 9200, 7200, 5100

• Lenovo Storage V Series™

• VersaStack™

With the use of IBM SVC almost every SAN attached storage system

- IBM as well as non-IBM - can be used. See capability matrix of IBM SVC.

http://www-01.ibm.com/support/docview.wss?uid=ssg1S4001497



Supported HANA configuration

The FlashCopy tool (FCtool) runs on SAP HANA TDI based on SUSE® SLES® or Red Hat®

-- on Intel® and IBM POWER™ LE platforms; for HANA scale-up and HANA scale-out.

Minimum version is HANA 2.0 SPS 4

Limitations by SAP

SAP HANA 2.0 up to SPS 3 only support HANA SnapShot in single tenant mode, mean-

ing one single application tenant in addition to the SYSTEM tenant. If more than one

application tenant has been configured, SAP HANA returns an error if you create a

HANA SnapShot, see SAP notes 2096000, 2039883.

Full HANA SnapShot support for real multi-tenant / MDC has been introduced with SAP

HANA 2.0 SPS 4.

SAP HANA Dynamic Tiering does not support HANA SnapShot, see SAP note 2375865,

even not with HANA 2.0 SPS 4.

VMware

SAP HANA can be run on a VMware® VM, the storage volumes (LUN) must be attached

as physical RDM; nothing else needs to be configured.

LINUX LVM

On each HANA system, setup multipath, PV, VG, and LV (physical volume, volume

group, logical volume) according to the IBM System Storage Architecture and Configu-

ration Guide for SAP HANA TDI

https://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP102347

and SAP note 1944799 - SAP HANA Guidelines for SLES Operating System Installation,

https://launchpad.support.sap.com/#/notes/1944799

HANA DATA and HANA LOG need their own VG (volume group).

There are no naming convention required for PV, VG, or LV, but the names may not

include the hyphen (minus) sign ‘-‘ .

ok: lvcreate 100%VG -n LV_DATA_1 VG_DATA_1

not: lvcreate 100%VG -n LV-DATA_1 VG-DATA_1

This FCtool does not require to use multipath alias, but you can use them.

Partitioning of PV / LUNs is not supported.


https://launchpad.support.sap.com/#/notes/1944799



Installation of IBM Spectrum Protect for ERP

On each HANA system, a proper backup solution must be installed, please read the

documentation:

Installation, configuration, and usage of IBM Spectrum Protect for Enterprise Re-

source Planning:

https://www.ibm.com/support/knowledgecenter/SSER83_8.1.9/erp.common/welcome.html

After the installation, you need to configure the DATA and LOG backup for the HANA

data base through the SAP HANA Studio:

• Select the HANA system in the System view

• Select Backup → Configuration → Backint in menu Log Backup Setting

Follow the procedure.

ssh communication

The FCtool needs to execute

system commands on remote

hosts; to do so, ssh keys need

to be setup on the FCtool par-

tition, and only on the FCtool

partition.

There is no need to ssh from

any HANA system to the FCtool

partition. In addition, ssh keys

needs to be setup only on the

FCtool partition for communi-

cation to the IBM Storage Sys-

tem.



FCtool deployment

It is required to install this FCtool on a separate LINUX VM/LPAR (with no HANA sys-

tem). This LINUX instance must have IP connections to all HANA systems, as well as

to the IBM Storage Systems. The HANA systems do not need to have an IP connection

to the storage system.

HANA rapid cold start -- SW RAID

IBM provides a solution to enable a fast start of SAP HANA, see these Solution Briefs:

https://www.ibm.com/downloads/cas/WQDZWBYJ

https://www.ibm.com/downloads/cas/1EJBRJXK

FCtool requires the following setup:

• The NVMe device must be configured as SW

raid

• The PVs must be configured as one VG with

one LV, similar like for standard DATA setup,

just without file system.

• The SW raid combining NVMe and VG must

defined as /dev/mdX, X:0..9

• Both devices (NVMe & VG) must be in state

"active sync"

• Only the VG device must be in state "writemostly"

• Define file system xfs on the DATA SW Raid, /dev/md1 in the picture

FCtool will just FlashCopy the LUNs and ignoring NVME.

https://www.ibm.com/downloads/cas/WQDZWBYJ

https://www.ibm.com/downloads/cas/1EJBRJXK



These are the steps FCtool executes during restore, to invalidate the data on NVMe:

1. umount /hana/data/sid

2. mdadm /dev/md1 --fail NVMe

3. mdadm /dev/md1 --remove NVMe

4. mdadm --stop /dev/md1

5. vgexport VG

6. FlashRestore

7. vgimport VG

8. mdadm --zero-superblock NVMe

9. mdadm --assemble --run /dev/md1 VG

10.mdadm /dev/md1 --add NVMe

11. mount /dev/md1 /hana/data/sid

NVMe: /dev/md0

VG : /dev/mapper/vg_data-lv_data

Support for Spectrum Virtualize HyperSwap

Support has been added to FCtool with these requirements:

Both sites (master/aux) must be online for backup or restore.

Spectrum Virtualize does not allow a HyperSwap volume to be a FlashCopy target,

therefore FCtool removes the volume copy from the remote (aux) site prior FlashCopy

and adds it backup after restore: the option --wait is required; if not set, volume copy

will not be re-added during restore.

FCtool cannot create the FlashCopy targets as HyperSwap VDisks. Again, Spectrum

Virtualize does not allow a HyperSwap volume to be a FlashCopy Target.

IMPORTANT: VD requirements

The maximal length of the VDISKs used for HANA DATA or HANA LOG is 32 character.

It is not allowed to use VDISK from HANA DATA and LOG as flashcopy source for any

other mapping, like for additional backup. Doing so will result in an FCtool error. If you

need an additional mapping (backup) use fctool: see fctool --backup --extra



Installation of FCtool

First time installation

To install FCtool these four steps need to be done:

1 Preparation of FCtool VM

2 Installing FCtool

3 Preparation of HANA VMs, chapter 7 13 Adding HANA systems to FCtool

4 Configuring of FCtool, chapter 8 Configuring FCtool

Preparation of FCtool VM

FCtool will be executed by a non-root user. This user should be used only for FCtool,

default name is fcuser. FCtool uses agents (on HANA VMs) via ssh communication,

to do so fcuser needs password-free ssh communication, gained via ssh-keys.

1) Create FCtool user fcuser on FCtool VM.

2) On FCtool VM as user fcuser create ssh keys:

ssh-keygen -t rsa

3) Copy ssh keys to itself, this FCtool VM:

ssh-copy-id FCtool_VM

FCtool installation

Download the package from ibm.com server, e.g. to your FCtool VM:

wget -nv https://public.dhe.ibm.com/storage/FCtool/FCtool_4.3.1.tar

This package includes the executables for IBM POWER and Intel based HANA sys-

tems.

Extract all files into directory e.g. /tmp

Read README.first.txt

To install the FCtool run:

./install help

./install fctool [--user:name]

This will create installation directory /opt/ibm/fctool and copy the platform dependent

files as well as the config files into it. You can specify another user than fcuser with

optional parameter --user

https://public.dhe.ibm.com/storage/FCtool/FCtool_4.3.1.tar



Manual installation of FCtool:

There may be reasons to do so ...

On FCtool VM:

1) Create directory /opt/ibm/fctool

2) Extract all files into directory /opt/ibm/fctool

3) In directory /opt/ibm/fctool, depending on your platform ppc or x86 copy:

cp fctool.ppc /usr/local/bin/fctool cp fcagent.ppc /usr/local/bin/fcuser

4) copy file fcuser.sudo to /etc/sudoers.d/fcuser

cp fcuser.sudo /etc/sudoers.d/fcuser

If you not use user fcuser, but e.g. jon:

in file fcuser.sudo, change line

fcuser ALL=NOPASSWD:/usr/local/bin/fcagent

to

jon ALL=NOPASSWD:/usr/local/bin/fcagent cp fcuser.sudo /etc/sudoers.d/jon

On HANA VM:

1) Depending on your platform ppc or x86 copy:

cp fcagent.ppc /usr/local/bin/fcuser

2) copy file fcuser.sudo to /etc/sudoers.d/fcuser

cp fcuser.sudo /etc/sudoers.d/fcuser

If you not use user fcuser, but e.g. jon:

in file fcuser.sudo, change line

fcuser ALL=NOPASSWD:/usr/local/bin/fcagent

to

jon ALL=NOPASSWD:/usr/local/bin/fcagent

cp fcuser.sudo /etc/sudoers.d/jon



Update of FCtool

Download the package from ibm.com server to your FCtool VM:

wget -nv https://public.dhe.ibm.com/storage/FCtool/FCtool_4.3.1.tar

Extract all files into directory e.g. /tmp

Run:

./install fctool [--user:name]

You are done.

There is no need to update fcagent manually, this is done by FCtool automatically.

Manual update, required only on FCtool VM:

1) Downlod package

2) Extract all files into directory /opt/ibm/fctool

3) In directory /opt/ibm/fctool, depending on your platform ppc or x86 copy:

cp fctool.ppc /usr/local/bin/fctool cp fcagent.ppc /usr/local/bin/fcuser

Done ...

Adding HANA systems to FCtool

The following steps are required in if you want to add a HANA system to FCtool.

Preparation on HANA VM

1) Create FCtool fcuser, member of groups sapsys,users

2) On FCtool VM as fcuser (or defined by --user) : copy the ssh key from the

FCtool VM to the HANA VM:

ssh-copy-id -i ~/.ssh/id_rsa.pub fcuser@HANA_VM

3) On HANA VM install fcagent (if not already done):

download FCtool package, extract all files to e.g. /tmp, and run

./install fcagent [--user:name]

or run from FCtool VM:

./install fcagent --host:HANA_VM [--user:name]

This will copy the fcagent to directory /usr/local/bin, and setup sudo

for fcuser (or --user).



Setup hdbuserstore

See SAP documents on how to create a dedicated database account for running back-

ups of your SAP HANA Database: "SAP HANA Overall Security Guide".

FCtool needs hdbuserstore keys with the appropriated privileges, for each tenant one

unique key.

If you do not want to use user SYSTEM, then create a user eg. BACKUPUSER with these

privileges:

BACKUP ADMIN

BACKUP OPERATOR

CATALOG READ

To create a key for HANA/MDC systems, run the command hdbuserstore, usually it is

in /usr/sap/<SID>/SYS/exe/hdb/

As fcuser user on each HANA system, for each tenant:

hdbuserstore SET <KEY> <ENVIRONMENT> <USER> <PASSWORD>

SYSTEM : hdbuserstore SET 001 node1:30013 BACKUPUSER <password>

SID : hdbuserstore SET 002 node1:30015 BACKUPUSER <password>

tenant 1: hdbuserstore SET 003 node1:30xxx BACKUPUSER <password>

tenant 2: hdbuserstore SET 004 node1:30xxx BACKUPUSER <password>

...

The individual port of each tenant can be identified with HANA Studio or HANA Cock-

pit.

To see the details of a given key, run:

hdbuserstore LIST <KEY>

For help run:

hdbuserstore -h

These hdb user-store keys are used by FCtool, and must be added in the file

fctool.ini, see chapter 8 Configuring FCtool



Configuring FCtool

To configure FCtool you only need to edit the file fctool.ini.

setup of fctool.ini

The ini file is structured via sections, the section name is surrounded with squared

brackets. Syntax: Parameter = Value

Multiple values are separated by comma. Only parameter HOST and

USER_STORE_KEY are multi value fields.

e.g:

HOST = node1,node2,node3

All keywords must be typed in uppercase.

Here an example ini file, the '#' sign indicates a comment.

[ALL]

LOG_PRE = fctool

LOG_POST = SID TASK

LOG_DIR = /var/log/

[STORAGE]

TYPE = SVC

HOST = v7000

USER = fcadmin

# optional, for cmd: --copy

POOL_DATA = Poolname

POOL_LOG = Poolname

[HANA PR1]

SID = PR1

INSTANCE = 00

USER_STORE_KEY = 001,002,003

GLOBAL.INI = DEFAULT

HOST = plnxpr1

COM_ENCRYPTION_HDB = -e -ssltrustcert

BACKUP_VERSIONS = 10


POOL_LOG = Poolname

# optional: for cmd --remote

REMOTE_CMD = /hana/shared/process.sh

# optional

PRE_CMD_COPY = /opt/ibm/fctool/pre_cmd_copy.sh

POST_CMD_COPY = /opt/ibm/fctool/post_cmd_copy.sh

COM_TIME_OUT_HDB = 300

[HANA DT1]

SID = DT1

INSTANCE = 05

USER_STORE_KEY = key1,key2

GLOBAL.INI = DEFAULT

HOST = plnxdt1


# optional if this SID is used TARGET SID for cmd --refresh DT1

PRE_CMD_REFRESH = /opt/ibm/fctool/pre_cmd_refresh.sh

POST_CMD_REFTRSH = /opt/ibm/fctool/post_cmd_refresh.sh

COM_TIME_OUT_PRE_POST = 2000

# section [REMOTE] is optional, for cmd --remote

[REMOTE]

HOST = plnxbackup

# optional


POOL_LOG = Poolname



Sections and parameters are mandatory, else the parameter is marked with a *

Section Parameter Value

[ALL]

LOG_PRE *

LOG_POST*

PREfix of the log file name, default “FCtool”.

POSTfix of the log file name, default empty.

can be set to SID and/or TASK.

File names have this schema:

PRE.YYYY.MM.DD-hh:mm:ss.POST.log

e.g.

LOG_PRE and LOG_POST not set:

FCtool.YYYY.MM.DD-hh:mm:ss.log

LOG_PRE = fctool

LOG_POST = SID

fctool.YYYY.MM.DD-hh:mm:ss.ANA.log

LOG_PRE = john

LOG_POST = SID TASK

john.YYYY.MM.DD-hh:mm:ss.ANA.backup.log

LOG_DIR * Full qualified path, default: /var/log/FCtool/

LOG_TIME_STAMP * Specifies the time stamp format in debug output: RELATIVE: default, displays elapsed time in milli sec-

onds since start for each process step ABSOLUTE_SHORT: displays the absolute time for each process

step ABSOLUTE_LONG: displays the absolute time and date for each

process step.

[STORAGE]

TYPE Only Spectrum Virtualize is supported, entry: SVC

HOST hostname of the storage system

USER The user defined on SVC/Storwize

COPY_RATE * The copyrate for SVC/Storwize used by restore, copy, clone. Can be specified between 100 and 150, default 150.

CLEAN_RATE * The cleanrate for SVC/Storwize used by backup, restore, copy, clone. Can be specified between 100 and 150, default 150. See chapter 10 for details.

POOL_DATA *

POOL_LOG *

The pool names where to create the volumes in, for --copy. Default: The primary storage pool used for the DATA / LOG Volumes from [HANA SID].

[HANA <SID>]

SID Specifies the SID of the HANA system. The value must be placed in the section name as well

INSTANCE Specifies the instance number

USER_STORE_KEY Specifies the keys as defined with hdbuserstore, First entry must be for SYSTEM, second for SID, followed by all ten-ants, e.g. SYSTEM:001, SID:002, tenant1:003 001,002,003



GLOBAL.INI Full qualified path and name for global.ini or DEFAULT. Hint: There are multiple global.ini within one single HANA system, find the one holding the parameter basepath_datavolumes and basepath_logvolumes You can specify DEFAULT if scale up, and mount points are /hana/data/<SID> and /hana/log/<SID> e.g.: /hana/data/ANA /hana/log/ANA Not: /hana/data /hana/log

HOST The host(s) this HANA systems runs on, could be fully qualified do-main name

COM_ENCRYPTION_HDB* Specifies how to encrypt the hdbsql session To get started please see SAP knowledge center. Blog: https://blogs.sap.com/2018/08/06/connecting-to-an-sap-hana-service-database/ e.g.: -e -ssltrustcert

BACKUP_ VERSIONS

Number of FlashCopy backups you want to keep. The resulting >Target Volumes< volumes will be created automati-cally. HINT: You can increase or decrease this number during lifetime.

BACKUP_EXTRA_ VERSIONS

Number of Extra FlashCopy backups you want to keep. This type of backup is used for long-time backup and will be expired if number is reached and the extra backup is older than BACKUP_EXTRA_TIME. HINT: You can increase or decrease this number during lifetime.

BACKUP_EXTRA_ TIME

Specifies the minimum age in days for extra backup. HINT: You can increase or decrease this number during lifetime.

COPY_MODE * Specifies the copy mode for the TARGET volume (VDisk) Default: SPACEEFFICIENT, else FULLCOPY

CLEAN_MODE * Specifies if cleaning should be done with flashcopy maps, if COPY_MODE is set to SPECEEFFICENT. Default: YES, else NO. See chapter 10 for details.

REMOTE_CMD * For command --remote The command will be executed on the HOST specified in the [REMOTE] section.

PRE_CMD_REFRESH * POST_CMD_REFRESH

These scripts will be executed before and after the cmd --refresh on the FCtool host. The parameters are taken from the [HANA TARGET_SID] entry.

PRE_CMD_COPY * POST_CMD_COPY

These scripts will be executed before and after the cmd --copy on the FCtool host. The parameters are taken from the [HANA SOURCE_SID] entry.

POOL_DATA * POOL_LOG

The pool names to create the Target Volumes in. If not set, the storage pool is used of the DATA / LOG Volumes from [HANA SID].

- Default: Primary pool of source VD is taken

- Option is set e.g: POOL_DATA = V7000_1 FlashCopy target is created in pool V7000_1

- Option is set e.g: POOL_DATA = V7000_1:V7000_2 FlashCopy target is created in pool V7000_1, and a copy in pool V7000_2

The separator between the two pools has to be the colon sign ":"

- Option is set e.g: POOL_DATA = COPY For VDisk mirror only, not for HyperSwap FlashCopy target is created in primary and copy Pool of the source VD, if primary has a copy -- else just a single copy in the primary pool.

https://blogs.sap.com/2018/08/06/connecting-to-an-sap-hana-service-database/

https://blogs.sap.com/2018/08/06/connecting-to-an-sap-hana-service-database/



POOL_DATA_UTIL *

POOL_LOG_UTIL Max allowed utilization of target pools for backup and copy opera-tion, default 90%. If utilization is higher, FCtool will not perform backup or copy operation and returns an error.

COM_TIME_OUT_HDB * Default 120 seconds, specifies the com time out for HDB com-mands in seconds.

COM_TIME_OUT_

PRE_POST *

Default 1200 seconds, specifies the com time out for PRE and POST commands in seconds.

[REMOTE] *

HOST The host to mount the latest FlashCopy Backup on, and to execute REMOTE_CMD

POOL_DATA * POOL_LOG

The pool names where to create the temporary volumes in. Default: The primary storage pool is used of the DATA / LOG Vol-umes from [HANA SID].



multiple fctool.ini files

With one fctool.ini file multiple HANA systems can be managed; but limited to one

storage systems. If you have more than one storage systems then you need individual

fctool.ini files for each storage system, to do so use the fctool option --ini filename.

For instance, you have two storage system V7000 and V5000, and the associated

fctool.ini files fctoolV7000.ini and fctoolV5000.ini, then you run e.g for backup:

fctool --backup --sid ANA --ini /ibm/opt/fctool/fctoolV7000.ini

fctool --backup --sid ERP --ini /ibm/opt/fctool/fctoolV5000.ini

Testing FCtool

To verify the HANA, LINUX, and Storage System setup, the FCtool provides a test fea-

ture. It is recommended to use this test sequence:

1 INI List context of fctool.ini

2 LVM Test if all data can be processed: MP→ LV→ VG→ PV→ UUID

3 LVM:umount Umount DATA and exports VG

4 LVM:mount imports VG and mounts DATA

5 STORAGE Connects to STORAGE & checks the required VD.

6 HDB Test connection to HANA

These tests can be used if remote operation will be used:

REMOTE:mount Creates temporary FlashCopy from last Backup of --sid,

and mounts it at host --host (DATA & LOG)

REMOTE:umount Umounts the temporary FlashCopy and deletes the Flash-

Copy (DATA & LOG), also from storage system

REMOTE:checkVG Check if old PV entries still exists, and if, deletes them

(DATA & LOG)

Syntax: fctool --test <WHAT> --sid <SID>

These are example output, they may change with new FCtool versions:

# fctool --test INI

ALL - LOG_DIR : /var/log/FCtool/

ALL - LOG_TIME_STAMP : RELATIVE

ALL - USER :

MF1 - BACKUP_EXTRA_TIME : 10

MF1 - BACKUP_EXTRA_VERSIONS : 3

MF1 - BACKUP_VERSIONS : 5

MF1 - CLEAN_MODE : NO

MF1 - COM_ENCRYPTION_HDB :

MF1 - COM_TIME_OUT_HDB : 120

MF1 - COM_TIME_OUT_PRE_POST : 1200



MF1 - COPY_MODE : SPACEEFFICIENT

MF1 - GLOBAL.INI : DEFAULT

MF1 - HOST : phmfn1

MF1 - INSTANCE : 00

MF1 - MAXSNAPSHOTS : 5

MF1 - POOL_DATA : FS900_01_child1

MF1 - POOL_LOG : FS900_01:V7000_02

MF1 - POST_CMD_COPY :

MF1 - POST_CMD_REFRESH :

MF1 - PRE_CMD_COPY :

MF1 - PRE_CMD_REFRESH :

MF1 - REMOTE_CMD : /tmp/remote.cmd

MF1 - SID : MF1

MF1 - USER_STORE_KEY : 001,002

MF2 - BACKUP_EXTRA_TIME : 7

MF2 - BACKUP_EXTRA_VERSIONS : 10

MF2 - BACKUP_VERSIONS : 3

MF2 - CLEAN_MODE : YES

MF2 - COM_ENCRYPTION_HDB :

MF2 - COM_TIME_OUT_HDB : 120

MF2 - COM_TIME_OUT_PRE_POST : 1200

MF2 - COPY_MODE : SPACEEFFICIENT

MF2 - GLOBAL.INI : default

MF2 - HOST : phmfn2

MF2 - INSTANCE : 00

MF2 - MAXSNAPSHOTS : 3

MF2 - POOL_DATA : Flash

MF2 - POOL_LOG : Flash

MF2 - POST_CMD_COPY :

MF2 - POST_CMD_REFRESH :

MF2 - PRE_CMD_COPY :

MF2 - PRE_CMD_REFRESH :

MF2 - REMOTE_CMD : /root/fc_remote.sh

MF2 - SID : MF2

MF2 - USER_STORE_KEY : 001

REMOTE - HOST : phmfn3

REMOTE - POOL_DATA :

REMOTE - POOL_LOG :

STORAGE - CLEAN_RATE : 100

STORAGE - COPY_RATE : 100

STORAGE - HOST : v7000

STORAGE - POOL_DATA :

STORAGE - POOL_LOG :

STORAGE - TYPE : SVC

STORAGE - USER : fcuser

fctool --test LVM --sid H09

FCTSTART Version: 4.1.0 Start time: 2018.08.21 14:08:52

FCT3600I Starting LVM

Active Host : ph09na1

Mount Point DATA : /hana/data/H09

Logical Volume : h09datalv

Volume Group : h09datavg

Physical Volumes : h09data1,h09data2,h09data3,h09data4

Mount Point LOG : /hana/log/H09

Logical Volume : h09loglv

Volume Group : h09logvg

Physical Volumes : 3600507640081811fe80000000000242b, 3600507640081811fe80000000000242c,

3600507640081811fe80000000000242d,3600507640081811fe80000000000242e

FCT3699I Test: LVM completed successful.

fctool --test STORAGE --sid H09


FCT3600I Starting STORAGE

DATA PV:h09data1, VD:volume-ph09na1-h09data1-c3da9366-b091

DATA PV:h09data2, VD:volume-ph09na1-h09data2-1c0f0d84-4265



DATA PV:h09data3, VD:volume-ph09na1-h09data3-0380ba03-f0eb

DATA PV:h09data4, VD:volume-ph09na1-h09data4-aefa2675-d363

LOG PV:3600507640081811fe80000000000242b, VD:volume-ph09na1-h09log1-1f75c014-cc7a

LOG PV:3600507640081811fe80000000000242c, VD:volume-ph09na1-h09log2-f271d5fd-8ffc

LOG PV:3600507640081811fe80000000000242d, VD:volume-ph09na1-h09log3-2ea80ed9-d102

LOG PV:3600507640081811fe80000000000242e, VD:volume-ph09na1-h09log4-299f91ac-132a

FCT3699I Test: STORAGE completed successful.

fctool --test HDB --sid H09


FCT3600I Starting HDB

FCT3612I Connected to host:ph09na1

FCT3699I Test: HDB completed successful.

FCTEND successful, run time: 0 seconds.

$ fctool --test REMOTE:mount --host ph09na3 --sid H09


FCT3600I Starting REMOTE mount

FCT3455I Verifying MP on target host:ph09na3

FCT3456I Creating VD for ph09na3

FCT3457I Starting FlashCopy from Target:FC1534780707544

FCT7654I Creating CG -autodelete -name H09_ph09na3_COPY

FCT7660I Creating FlashCopy Mappings ...

FCT3460I Scanning host ph09na3 for update PV & VG.

FCT3461I Mounting new file systems on host ph09na3:/hana/data/H09,/hana/log/H09

FCT6311W mount command requires file-system-check


FCT3699I Test: REMOTE mount completed successful.


fctool --test REMOTE:umount --host ph09na3 --sid H09


FCT3600I Starting REMOTE umount

FCT3701I Starting removing MP & VG from host ph09na3, sid:H09

FCT3702I Unmounting file systems.

FCT3703I Removing PV from host.

FCT3706I Removing VD from Storage System.

FCT3712I Removing VG:h09logvg,h09logvg on host ph09na3 completed successfully.

FCT3699I Test: REMOTE umount completed successful.




Using FCtool

This section provides information how to use the FCtool, to start it, run:

fctool

fctool --<command> --sid <SID> [--debug 1..5] [--batch] [--wait|--force]

fctool --version

fctool -? | --help | --man

Displays command syntax or full man pages

--debug 1..5

Option debug specifies the debug (trace) level from 1 to 5 for screen output

1 provides base process flow , while 5 provided very detailed information.

For tracing level 3 or 4 are useful.

A log file is generated always, with debug level 5

--debug 1 is the default level

--batch

All screen output is sent to log file. This option is required if FCtool is run in

batch mode, e.g via a scheduler; only valid for --backup.

--wait

If a previous restore process has not been completed on SVC, a follow on next

restore will fail, unless you specify --wait. In this case FCtool waits until all pre-

cious tasks have been completed.

Needs to be specified for HyperSwap volumes as well. The copy-volume needs

to be removed before FlashCopy restore can be performed, and the volume

copy can only be re-added after the FlashCopy has been completed.

With --wait FCtool waits for completion, and then adds back the volume-copy.

--force

If a previous refresh process has not been completed on SVC, a follow on next

restore will fail, unless you specify --force. In this case FCtool stops the current

process and starts a new.

--ini file

specifies the ini file to be used, if not default /opt/ibm/fctool/fctool.ini

Specify full path name



Supported functions with FCtool V4:

functions IBM Power VM / Native Intel VMware ESXi with pRDM

HANA scale up HANA scale out HANA scale up HANA scale out

backup yes yes yes yes

restore yes yes yes yes

remote yes yes no no

copy yes no no no

refresh yes no no no

SW RAID yes1) no no no

1) Power only, backup/restore only

Backup

--backup --sid <SID> [--extra] [--batch]

This command creates a new (standard) FlashCopy backup, if --extra is specified it

performs an EXTRA FC backup:

fctool -b -s H09


FCT3101I Starting FlashCopy Backup for HANA:H09

FCT3102I Checking Storage Volumes

FCT3103I New HANA Snapshot ID:1535022274587

FCT3104I Preparing Backup

FCT7204I New FlashCopy Target:FC1535022274587

FCT7214I Creating VDs for FC1535022274587

FCT7230I Creating DATA CG:CG_H09_FC1535022274587

FCT7233I Creating LOG CG:CG_H09_FC1535022274587_LOG

FCT7235I Creating FC Maps for FC1535022274587

FCT3105I Executing FlashCopy Backup

FCT7270I Starting CG:CG_H09_FC1535022274587

FCT7272I Starting CG:CG_H09_FC1535022274587_LOG

FCT3106I FlashCopy Backup executed successfully, target used:FC1535022274587 for

HANA Snapshot:1535022274587

FCT3110I Starting FlashCopy post process

FCT7360I Processing FC:1535015011919 Number of current FC:3, goal:2

FCT7390I Goal reached: Number of existing FC is less or equal number of requested, ending

FCT3198I FlashCopy Backup processing completed


If you want to keep one FlashCopy per day for one week, then you need to set

BACKUP_VERSIONS = 7

If you want to keep one FlashCopy Backup each hour for one day, then:




After the nth+1 backup has been performed, the oldest one will be deleted.

If you want to keep a maximum of 10 EXTRA FlashCopies for at least 30 days, then

then you need to set:

BACKUP_EXTRA_VERSIONS = 10

BACKUP_EXTRA_TIME = 30

Note: By default, the standard FlashCopy backup is space efficient, and FC Map-

pings as well as FC Consistency Groups will be kept until expiration.

An Extra Backup is always a full copy, and FC mappings and FC Consistency

Groups are deleted after background copy has been finished.

Note: EXTRA backup does not create a HANA SnapShot. HANA does not treat this

backup as SnapShot backup, no recover process is required by HANA, just a

HANA restart.

Note: Older EXTRA backup will expire only if both criteria are met:

1) Number of extra backup exceeded

2) Age has been reached for oldest backup:

If you have set BACKUP_EXTRA_VERSIONS=5 and BACKUP_EXTRA_TIME=10 and you created

one extra backup each day, then after the 6th backup has been taken the old-

est backup will not be deleted, because it is only 6 days old.

After taking the 11th backup the oldest backup will be deleted, because now

this backup is older than 10 days.

Restore

--restore <FlashCopy> --sid <SID> [--wait]

Performs a restore from the given <FlashCopy>. The FlashCopy ID can be identified

via the --query command.

Note: Only the DATA volumes are restored. The LOG data are handled by HANA via the

BACKINT interface, via IBM Spectrum Protect for ERP. This allows to roll-forward

HANA after FlashCopy restore.

fctool -r FC1534845222472 -s H10


FCT3201I Starting FlashCopy Restore for HANA:H10

FCT3204I Umounting file system:/hana/data/H10

FCT3205I Start FlashRestore from target:FC1534845222472

FCT7504I Creating CG -autodelete -name CG_H10_FC1534845222472_R

FCT7521I Starting CG -restore:CG_H10_FC1534845222472_R

FCT3210I Mounting file system:/hana/data/H10

FCT3298I FlashCopy Restore for target:FC1534845222472 completed successfully




Note: The HANA system must be offline. The FCtool does not stop nor start an SAP

HANA system. After a FlashCopy restore you need to recover HANA via the SAP

STUDIO, see chapter: 12 Recover the SAP HANA system.

Note: If you restore an EXTRA backup then HANA needs just to be restarted. No HANA

recovery is necessary. With extra, LOG will be restored as well to the exact same

point-in-time as DATA.

Note: The reverse FlashCopy process needs to be completed before another restore

or backup task can start; this is ensured by the FCtool. You can use the --wait

option, then FCtool waits until the previous restore has been completed.

Note: The reverse FlashCopy process needs to be completed before a volume copy

can be added for HyperSwap; this is ensured by the FCtool. You can use the

--wait option, then FCtool waits until restore has been completed and then

adding back the volume copy; else restore completes without adding back the

volume copy.

Hint: After the restore has completed another restore from the same or another

TARGET can be performed.

Backup and Restore of MDC

During backup of a multi-tenant system, the entire HANA system is backed up, and

vise-versa, if you restore a multi-tenant system then the entire HANA system is re-

stored, incl SYSTEM, SID, and all additional tenants.

Each tenant must be recovered individually, see SAP documentation.

SAP HANA does not allow to restore a single tenant backed up via SNAPSHOT: This is

a limitation of HANA (SNAPSHOT), not of FCtool or FlashCopy.

Remote

--remote --sid <SID>

This command mounts the latest FlashCopy at the remote host (DATA & LOG), starts

the remote command as specified with REMOTE_CMD, and unmounts the file systems

afterwards. The FCtool creates temporary Storage Volumes and maps the latest Target

to them.

The remote command can be specified like this:

REMOTE_CMD = /my_directory/myscript.sh $SID $DATA $LOG

$SID, $DATA, and $LOG are optional variables, which will be evaluated (expanded) to

current values during execution, e.g. to:



/my_directory/myscript.sh ANA /hana/data/ANA/mnt0001 /hana/log/ANA/mnt00001

Even if the remote cmd finishes with an error, the remote host will be "cleaned up",

with means all attached disk (PV) and VG will be removed, and the temporary Storage

Volumes (VD) will be deleted.

The temporary VD will be created in the Storage pool as defined by parameters

POOL_DATA and POOL_LOG in section [REMOTE]. If these parameters are not set, the

VDs will be created in the same pool of the Backup Volumes.

Note: If you get this error msg:

ERROR ... VG vg_data_1 already exists on <host_name>

then most likely the cleanup of a previously --remote command did not complete

successfully.

To cleanup this mount run:

--test REMOTE:umount --host <host_name> --sid <SID>

Note: The --remote command will be called as user fcuser, if you need other cre-

dentials, then the commands need to be executed via sudo, eg. in fc_tool.ini:

REMOTE_CMD = sudo /hana/shadred/my_script.sh --sid $SID --data $DATA --log $LOG

Query

--query --sid <SID>

This command prints out the current existing HANA SNAPSHOTS with the associated

IBM FlashCopy TARGETS. If HANA is online, it connects to it, and pulls the existing

information from HANA. If HANA is off-line, it displays the stored information.

Note: If no FlashCopy backup exists, this function returns an error.

# fctool --query --sid H09

HANA BACKUP ID FlashCopy Target Status Comment Time Completed

1534780707544 FC1534780707544 successful IBM-FlashCopy-Backup 2018-08-20 17:58:45

1534780456772 FC1534780456772 successful IBM-FlashCopy-Backup 2018-08-20 17:54:34

--- EX1534855285000 Storage Extra 2018-08-21 14:41:25

--- EX1534855179000 Storage Extra 2018-08-21 14:39:39



HANA Studio screenshot sample:

Note: After you have restored a HANA system, the list of available HANA snapshots

and Storage FlashCopy are out of sync.

If you want to list the availed FlashCopy taken after the point-in-time of the restored

Snapshot then issue this command:

fctool --query storage --sid SID

If you want to list SnapShots known by HANA issue this command:

fctool --query HDB --sid SID

Only the listed (available) FlashCopy on the storage system can be restored, inde-

pendent of the list of known SnapShots.

Audit Storage

--test STORAGE:FCMAP --sid <SID>

It may happen that a VD goes offline, a FC mapping got “stuck” or something else went

wrong ...

This command shows all used VD and target VD, as well as the associated FC map-

pings and CG for this SID, as well as their states.

EBID: External Backup ID

Here with IBM FlashCopy TARGET

name



fctool --test STORAGE:FCMAP --sid MF1


FCT1104I Log file: /var/log/FCtool/FCtool.2020.01.07-15:04:18.log

FCT3600I Starting STORAGE FCMAP

DATA PV:3600507640081811fe800000000004422, VD:volume-phmfn1-mf1data1-f3c61db0-5c70

DATA PV:3600507640081811fe800000000004423, VD:volume-phmfn1-mf1data2-bcbbacfa-a05d

DATA PV:3600507640081811fe800000000004424, VD:volume-phmfn1-mf1data4-86a29c63-6c0a

DATA PV:3600507640081811fe800000000004425, VD:volume-phmfn1-mf1data3-752e4aec-6585

LOG PV:3600507640081811fe80000000000441e, VD:volume-phmfn1-mf1log1-ba120227-d6b0

LOG PV:3600507640081811fe80000000000441f, VD:volume-phmfn1-mf1log2-d1b791ff-4b6c

LOG PV:3600507640081811fe800000000004420, VD:volume-phmfn1-mf1log3-ffd458b3-12bd

LOG PV:3600507640081811fe800000000004421, VD:volume-phmfn1-mf1log4-48c40619-f2de

VD :volume-phmfn1-mf1data1-f3c61db0-5c70

target:volume-phmfn1-mf1data1-f3c61db0-5c70_FC1578235546000,volume-phmfn1-mf1data1-f3c61db0-5c70_FC1578394591784,

volume-phmfn1-mf1data1-f3c61db0-5c70_FC1578320765000,volume-phmfn1-mf1data1-f3c61db0-5c70_FC1578321171000,

volume-phmfn1-mf1data1-f3c61db0-5c70_FC1578321385000,

volume-phmfn1-mf1data1-f3c61db0-5c70_FC1578321385000,

FCmap :volume-phmfn1-mf1data1-f3c61db0-5c70_FC1578235546000_MAP,CG_MF1_FC1578235546000,

volume-phmfn1-mf1data1-f3c61db0-5c70_FC1578394591784_MAP,CG_MF1_FC1578394591784,




VD :volume-phmfn1-mf1data2-bcbbacfa-a05d

target:volume-phmfn1-mf1data2-bcbbacfa-a05d_FC1578235546000,volume-phmfn1-mf1data2-bcbbacfa-a05d_FC1578394591784,

volume-phmfn1-mf1data2-bcbbacfa-a05d_FC1578320765000,volume-phmfn1-mf1data2-bcbbacfa-a05d_FC1578321171000,

volume-phmfn1-mf1data2-bcbbacfa-a05d_FC1578321385000,

volume-phmfn1-mf1data2-bcbbacfa-a05d_FC1578321385000,

FCmap :volume-phmfn1-mf1data2-bcbbacfa-a05d_FC1578235546000_MAP,CG_MF1_FC1578235546000,

volume-phmfn1-mf1data2-bcbbacfa-a05d_FC1578394591784_MAP,CG_MF1_FC1578394591784,




VD :volume-phmfn1-mf1data4-86a29c63-6c0a

target:volume-phmfn1-mf1data4-86a29c63-6c0a_FC1578235546000,volume-phmfn1-mf1data4-86a29c63-6c0a_FC1578320765000,

volume-phmfn1-mf1data4-86a29c63-6c0a_FC1578394591784,volume-phmfn1-mf1data4-86a29c63-6c0a_FC1578321171000,

volume-phmfn1-mf1data4-86a29c63-6c0a_FC1578321385000,

volume-phmfn1-mf1data4-86a29c63-6c0a_FC1578321385000,

FCmap :volume-phmfn1-mf1data4-86a29c63-6c0a_FC1578235546000_MAP,CG_MF1_FC1578235546000,

volume-phmfn1-mf1data4-86a29c63-6c0a_FC1578320765000_MAP,CG_MF1_FC1578320765000,




VD :volume-phmfn1-mf1data3-752e4aec-6585

target:volume-phmfn1-mf1data3-752e4aec-6585_FC1578235546000,volume-phmfn1-mf1data3-752e4aec-6585_FC1578320765000,

volume-phmfn1-mf1data3-752e4aec-6585_FC1578394591784,volume-phmfn1-mf1data3-752e4aec-6585_FC1578321171000,

volume-phmfn1-mf1data3-752e4aec-6585_FC1578321385000,

volume-phmfn1-mf1data3-752e4aec-6585_FC1578321385000,

FCmap :volume-phmfn1-mf1data3-752e4aec-6585_FC1578235546000_MAP,CG_MF1_FC1578235546000,

volume-phmfn1-mf1data3-752e4aec-6585_FC1578320765000_MAP,CG_MF1_FC1578320765000,




VD :volume-phmfn1-mf1log1-ba120227-d6b0

target:volume-phmfn1-mf1log1-ba120227-d6b0_FC1578235546000,volume-phmfn1-mf1log1-ba120227-d6b0_FC1578320765000,

volume-phmfn1-mf1log1-ba120227-d6b0_FC1578394591784,volume-phmfn1-mf1log1-ba120227-d6b0_FC1578321171000,

volume-phmfn1-mf1log1-ba120227-d6b0_FC1578321385000,

volume-phmfn1-mf1log1-ba120227-d6b0_FC1578321385000,

FCmap :volume-phmfn1-mf1log1-ba120227-d6b0_FC1578235546000_MAP,CG_MF1_FC1578235546000_LOG,

volume-phmfn1-mf1log1-ba120227-d6b0_FC1578320765000_MAP,CG_MF1_FC1578320765000_LOG,




VD :volume-phmfn1-mf1log2-d1b791ff-4b6c

target:volume-phmfn1-mf1log2-d1b791ff-4b6c_FC1578235546000,volume-phmfn1-mf1log2-d1b791ff-4b6c_FC1578320765000,

volume-phmfn1-mf1log2-d1b791ff-4b6c_FC1578394591784,volume-phmfn1-mf1log2-d1b791ff-4b6c_FC1578321171000,

volume-phmfn1-mf1log2-d1b791ff-4b6c_FC1578321385000,

volume-phmfn1-mf1log2-d1b791ff-4b6c_FC1578321385000,

FCmap :volume-phmfn1-mf1log2-d1b791ff-4b6c_FC1578235546000_MAP,CG_MF1_FC1578235546000_LOG,

volume-phmfn1-mf1log2-d1b791ff-4b6c_FC1578320765000_MAP,CG_MF1_FC1578320765000_LOG,




VD :volume-phmfn1-mf1log3-ffd458b3-12bd

target:volume-phmfn1-mf1log3-ffd458b3-12bd_FC1578235546000,volume-phmfn1-mf1log3-ffd458b3-12bd_FC1578394591784,

volume-phmfn1-mf1log3-ffd458b3-12bd_FC1578320765000,volume-phmfn1-mf1log3-ffd458b3-12bd_FC1578321171000,

volume-phmfn1-mf1log3-ffd458b3-12bd_FC1578321385000,

volume-phmfn1-mf1log3-ffd458b3-12bd_FC1578321385000,

FCmap :volume-phmfn1-mf1log3-ffd458b3-12bd_FC1578235546000_MAP,CG_MF1_FC1578235546000_LOG,

volume-phmfn1-mf1log3-ffd458b3-12bd_FC1578394591784_MAP,CG_MF1_FC1578394591784_LOG,






VD :volume-phmfn1-mf1log4-48c40619-f2de

target:volume-phmfn1-mf1log4-48c40619-f2de_FC1578235546000,volume-phmfn1-mf1log4-48c40619-f2de_FC1578394591784,

volume-phmfn1-mf1log4-48c40619-f2de_FC1578320765000,volume-phmfn1-mf1log4-48c40619-f2de_FC1578321171000,

volume-phmfn1-mf1log4-48c40619-f2de_FC1578321385000,

volume-phmfn1-mf1log4-48c40619-f2de_FC1578321385000,

FCmap :volume-phmfn1-mf1log4-48c40619-f2de_FC1578235546000_MAP,CG_MF1_FC1578235546000_LOG,

volume-phmfn1-mf1log4-48c40619-f2de_FC1578394591784_MAP,CG_MF1_FC1578394591784_LOG,




These CG are in use :CG_MF1_FC1578235546000,CG_MF1_FC1578320765000,CG_MF1_FC1578320765000_LOG,CG_MF1_FC1578321171000,

CG_MF1_FC1578321171000_LOG,CG_MF1_FC1578321385000,CG_MF1_FC1578321385000_LOG,CG_MF1_FC1578394591784,

CG_MF1_FC1578394591784_LOG,CG_MF1_FC1578235546000_LOG,

These CG are NOT in use:CG_MF1_FC1578058466000,CG_MF1_FC1578058466000_LOG,

FCT3699I Test: STORAGE FCMAP completed successful.


Refresh

--refresh <TARGET_SID> --sid <SOURCE_SID> [--force]

Performs a refresh of TARGET_SID DATA and LOG file systems from SOURCE_SID.

Both HANA systems (both SIDs) need to be configured in the ini file.

This command

1) If set, run PRE_CMD_REFRESH on FCtool host

2) Umount the TARGET file systems DATA & LOG

3) FlashCopy the SOURCE file systems over the TARGET file systems

4) Renames back the file systems from SOURCE to TARGET names

5) (re) mount the (new) TARGET file systems

6) if set, run POST_CMD_REFRESH on FCtool host

fctool --refresh H10 -s H09


FCT3501I Starting refresh of HANA:H10 from H09

FCT3555I Checking Storage Volumes

FCT3556I Umounting on host:ph09na2 -> MP:/hana/data/H10,/hana/log/H10

FCT3557I Checking if older Consistency Group still in process

FCT3579I Starting FlashCopy

FCT7654I Creating CG -autodelete -name H09_H10_COPY


FCT3572I Renaming VG & LV from h09datavg to h10datalv

FCT6503I Renaming VG h09datavg to h10datavg.

FCT6506I Renaming LV h09datalv to h10datalv -- VG:h10datavg

FCT3574I Renaming VG & LV from h09logvg to h10datalv

FCT6503I Renaming VG h09logvg to h10logvg.

FCT6506I Renaming LV h09loglv to h10loglv -- VG:h10logvg

FCT3575I Mounting file systems ...

FCT3548I Refresh of system H10 from source H09 completed successfully.




PRE_CMD _REFRESH / POST_CMD_REFRESH:

Both commands will be called with these parameters:

<command> --source <SID> --target <SID>

Specify full path name for <command>, e.g.

PRE_CMD_REFRESH = /hana/shared/refresh_pre.sh

If the PRE_CMD command return with error the refresh process will be stopped.

If the execution of FlashCopy fails, the POST_CMD will called with additional

parameter --error NO_FLASHCOPY

Note: The FCtool does not stop or start a SAP HANA system.

Note: After this REFRESH, you cannot start the TARGET system; you need to process

SAP's SAP HANA System Copy - Homogeneous Backup/Recovery Method. This com-

mand refreshes (overwrite via FlashCopy) the DATA & LOG file system only, not

/hana/shared/ or any other file system, nor configuration files.

➔ Can be automated via the Pre and Post commands.

Note: The commands will be called as user fcuser, if you need other credentials on

the SOURCE or TRAGET system, then the commands need to be executed via sudo,

and these commands (shell scripts) need to be added to sudores via visudo, e.g:

PRE_CMD_REFRESH = /opt/ibm/fctool/my_pre.sh

in /opt/ibm/fctool/my_pre.sh

Figure 1: Process flow for --refresh

#!/bin/bash

...

ssh source_host sudo /hana/shared/pre.sh

ssh target_host sudo /hana/shared/pre.sh

...



Note: The refresh FlashCopy process needs to be completed before another refresh

can start; you can use the --force option, then FCtool stops the background copy,

and restarts a new refresh process.

Hint: The group name as well as the logical volume name of SOURCE and TARGET

systems can be the same.

Hint: Create an EXTRA backup before you start the refresh, from the off-line HANA

system. This eases the restore process to the original state if the refresh process fails.

Note: The Refresh progress will run for some time, and the Consistency Group, as well

as the associated FC mappings may not be autodeleted, but stay in state coping with

100% progress. They will be deleted eventually if no other FC mapping refers to them,

but if the progress is at 100% it is safe to stop (only stop) the CG, then the associated

FC mappings as well as the CG will be auto-deleted.

Copy

--copy <TARGET_HOST> --sid <SOURCE_SID>

Performs a copy of SOURCE_SID DATA & LOG to TARGET_HOST.

The parameter TARGET_HOST specifies the host the HANA system is copied to.

fctool --copy ph09na3 --sid H09


FCT3401I Starting copy HANA H09 to host ph09na3

FCT3452I Starting copy of HANA:H09 to host ph09na3

FCT3455I Verifying MP on target host:ph09na3

FCT3456I Creating VD for ph09na3

FCT3457I Starting FlashCopy from Target:EX1534781116549

FCT7654I Creating CG -autodelete -name H09_ph09na3_COPY


FCT3460I Scanning host ph09na3 for update PV & VG.

FCT3461I Mounting new file systems on host ph09na3:/hana/data/H09,/hana/log/H09



FCT3498I Copy of system H09 to host ph09na3 completed successfully.

FCT3448I Copy of HANA H09 to host ph09na3 completed successfully.


Note: This command does only copy the HANA DATA & LOG file systems (VG), but not

/hana/shared/. Additional SAP processes are required to fully copy a HANA instance.

Note: First you need to install fcagent on the target host.

PRE_CMD_COPY / POST_CMD_COPY:

Both commands will be called on FCtool-VM as user fcuser, with these parameters:

<command> --source <SID> --data MP_DATA --log MP_LOG



Specify full path name for <command>, e.g.

PRE_CMD_COPY = /hana/shared/copy_pre.sh

If the PRE_CMD_COPY command return with error the refresh process will be

stopped.

Process flow is like –refresh

Note: The Copy progress will run for some time, and the Consistency Group, as well as

the associated FC mappings may not be autodeleted, but stay in state coping with

100% progress. They will be deleted eventually if no other FC mapping refers to them,

but if the progress is at 100% it is save to stop (only stop) the CG, then the associated

FC mappings as well as the CG will be auto-deleted.

How to delete FlashCopy backups

If you want to delete all backups of a HANA system/SID then you need to set in

fctool.ini:

BACKUP_VERSIONS = 0

BACKUP_EXTRA_VERSIONS = 0

and run:

fctool --backup --sid <SID> --force

fctool --backup --sid <SID> --force --extra

How FCtool uses FlashCopy Mappings of Spectrum Virtualize

FCtool creates the target volumes for the backup operation (dynamic target allocation)

and removes the oldest according fctool.ini parameter BACKUP_VERSIONS.

Please refer to the IBM documentation for Spectrum Virtualize, Storwize, or SVC how

the parameters copyrate and cleanrate are used by command mkfcmap.

FCtool provides four parameters to control the flashcopy behavior.

COPY_RATE is the equivalent for copyrate and can be set from 100 to 150.

A lower value than 100 is not useful. Default value is 150.

A value of 0 is a special case and will be handled by COPY_MODE

CLEAN_RATE is the equivalent for cleanrate and can be set from 100 to 150.

A lower value than 100 is not useful. Default value is 150.

A value of 0 is a special case and will be handled by CLEAN_MODE

COPY_MODE can be set to SPACEEFFICIENT (default) or FULLCOPY.

If set to SPACEEFFICIENT copyrate is set to 0,

else (if FULLCOPY) copyrate is set to COPY_RATE.



CLEAN_MODE can be set to YES or NO.

If set to NO and COPY_MODE is set to SPACEEFFICIENT then cleanrate is set to 0,

else cleanrate is set to CLEAN_RATE.

This table shows how the four parameters interacts for --backup:

--backup CLEAN_MODE

YES NO

COPY_MODE SPACE

EFFICIENT

copyrate=0

cleanrate=CLEAN_RATE

copyrate=0

cleanrate=0

FULLCOPY copyrate=COPY_RATE


copyrate=COPY_RATE


For --extra, --restore, --refresh, --copy the parameter cleanrate is set to

CLEAN_RATE, and copyrate is set to COPY_RATE.

Recommendation: To ease expiration of older backups use the default

CLEAN_MODE = YES

even this will lead to higher space consumption in the storage pools.

FCtool best practices / scheduling

Use IBM Spectrum Protect for ERP to backup DATA and LOG to the Spectrum Protect

(SP) server. Use the FCtool for additional FlashCopy protection, which allows you to

recover the DATA partitions in a few minutes rather than waiting on DATA to be re-

stored from SP server.

1. Setup LOG backup, specify the backup-interval to your needs.

Typically every 5 to 15 minutes

2. Setup DATA backup: Setup incremental interval to your needs, maybe once a

day, or even more often.

Specify Full backup to your needs, maybe once a week, or more often.

3. Schedule FCtool FlashCopy backup to your needs, maybe 4 times a day, or

more often.

Any scheduler can be used to start FCtool, then the option --batch needs to be set, e.g:

/opt/ibm/fctool/fctool --backup --sid ANA --batch



Recover the SAP HANA system

Please read the SAP HANA admin guide how to recover the HANA database.

You need to recover first the System DB, then the tenant DB.

Right-click on the offline HANA system →



Hint: List will be empty; STUDIO needs some time to display the snapshot list:



Second step, recover Tenant DB:



Confirm the following dialog -- Tenant is down, or will be stopped during recovery.



Resources

IBM System Storage Architecture and Configuration Guide for SAP HANA TDI


LINUX IO performance tuning for IBM System Storage


IBM SAN Volume Controller SVC (IBM SVC)

http://www-03.ibm.com/systems/storage/software/virtualization/svc/index.html

IBM Spectrum Protect for ERM, data protection for SAP HANA

https://www.ibm.com/support/knowledgecenter/SSER83/landing/welcome_sser83.html

SAP HANA Platform Admin Guide

SAP note:

1651055 - Scheduling SAP HANA Database Backups in Linux

2039883 - FAQ: SAP HANA database and storage snapshots

Service

IBM provides a service offering for installation, setup, and verification that it runs in

the appropriated environment. Please have your IBM representative or IBM Business

Partner send a request to [email protected]

Support & Help

If you have questions about the usage of FCtool, or you discovered an error, please

post your question at community.ibm.com, Middleware User Community, Global

Group Storage Management: https://ibm.biz/BdqBUP

Start your subject with fctool.

If you do not get the proper answer, then you can contact

[email protected]

Please include always the actual log file from /var/log/FCtool/, the files

fctool.ini, as well as the LVM.ini

Error msg

If you upgrade FCtool from V3 to V4, or you have increased the file-system size by

adding new PVs, then you will receive this error message:

FCT7355E lsvdiskdependentmaps …. CMMVC5754E The specified object does not

exist, or the name supplied does not meet the naming rules

or similar.



http://www-03.ibm.com/systems/storage/software/virtualization/svc/index.html

https://www.ibm.com/support/knowledgecenter/SSER83/landing/welcome_sser83.html


https://ibm.biz/BdqBUP




Then use this option until all V3 volumes or all volumes with old number of PVs have

been expired:

fctool -–ignore_error backup_post_object_does_not_exist --backup --sid SID

..

If you encounter another problem, please check the affected host with commands pvs

and multipath -ll.

On the Storage System check the Consistency Groups, FC Mapping, VDisks, they

should never be in state stopped or offline.

FCtool message codes

Explanation of messages FCtool displays: FCT0000X

FCT : Identifier (prefix) - will be displayed always

0000: Message number

X : Severity of message, high to low

F - Fatal error, means FCtool caught a program error

E - Error during execution

W - Warning, e.g. some limits reached, but can continue

I - Information

C - Process flow (Chase) (debug 2)

D - Debug information (debug 3)

T - Trace information (debug 4)

History

4.3.1 Added options LOG_PRE & LOG_POST in ini section [ALL]

FCtool now checks if required capacity is available in TARGET pool for backup and copy, see

POOL_DATA_UTIL and POOL_LOG_UTIL in ini [HANA] section.

FCtool now checks if VDISK of DATA and LOG have only valid flashcopy mappings, see chapter

5.11 IMPORTANT: VD requirements

4.3.0 Code cleanup: min HANA version is 2.0 SPS4, min Spectrum Virtualize version is 8.0.0,

Update from FCtool V3 not possible anymore

Add installation tool

Change: Only one package including IBM Power and Intel platform

Add ini parameter CLEAN_MODE and CLEAN_RATE

Add --test STORAGE:FCMAP to show all FC mapping , VD targets and their states

Fixed PRE/POST CMD, they were not executed on FCtool VM.

Due to sunset of IBM developerWorks location of forum has changed: https://ibm.biz/Bd2QLC

community.ibm.com => Milldleware User Community =>

Global Group Storage Management

4.2.6 Changed Spectrum Virtualize parameter cleanrate in FC mappings from 150 to 0 if copyrate is

0, used by fctool –backup with COPY_MODE = SPACEEFFCIENT

Add paragraph on how FCtool uses copyrate and cleanrate



4.2.5 Add ini parameter COPY_RATE in [STORAGE] section

4.2.4 New installation description

Now HANA systems running on POWER and on Intel can be managed by one FCtool instance.

4.2.3 Add HyperSwap support

Add description how to delete FlashCopy backups for a SID

4.2 Changes in USER_STORE_KEY

Add option COM_ENCRYPTION_HDB

4.1.d Add information about --ignore_error

Add information about HANA 2.0 SPS4 MDC support

4.1.c Add support for SW RAID, used by rapid cold start via NVMe

4.1 FCtool is delivered as compiled executable, no need to install perl/cpan modules anymore

Rename of modules: fc_tool.pl → fctool, fc_tool.ini → fctool.ini, fc_agent.pl → fcagent

New option --extra for --backup

Cleanup of PRE_CMD and POST_CMD for refresh and copy, will be executed on FCtool host

Trace file will be written always, with debug level 5

Package is located on https://public.dhe.ibm.com/storage/FCtool/

HTML GUI has been removed from the package

3.2 No need to update/install perl modules anymore

Added support for VDISK mirroring -- SVC stretched cluster

Added support for VMware

Added option --wait

Change parameter name: TRACE_LEVEL → LOG_LEVEL

Obsolete: LOG_FILE

3.1 FCtool does no longer need to run as root, now as regular user; new module fc_agent.pl

2.5 Change the way how to do backups: No predefined TARGET_SET will be used anymore, in-

stead for every new backup a new TARGET will be created, to avoid stopping a CG group

before executing the backup.

Also this enable the increase and decrease of backup version during lifetime.

2.4 Fix several minor errors.

Support for SLES 12.

2.3 Add security section: user & network access

Fixed naming schema of TARGET_SET.

2.2 Fixed support matrix, fix program error on scale out, added support matrix for functions,

added creation of target VD for backup.

2.1 Initial version based on perl

Notices

This information was developed for products and services offered in the U.S.A. IBM

may not offer the products, services, or features discussed in this document in other

countries. Consult your local IBM representative for information on the products and



services currently available in your area. Any reference to an IBM product, program, or

service is not intended to state or imply that only that IBM product, program, or service

may be used. Any functionally equivalent product, program, or service that does not

infringe any IBM intellectual property right may be used instead. However, it is the

user's responsibility to evaluate and verify the operation of any non-IBM product, pro-

gram, or service.

IBM may have patents or pending patent applications covering subject matter de-

scribed in this document. The furnishing of this document does not grant you any li-

cense to these patents. You can send license inquiries, in writing, to:

IBM Director of Licensing, IBM Corporation, North Castle Drive, Armonk, NY 10504-

1785 U.S.A.

The following paragraph does not apply to the United Kingdom or any other Country

where such provisions are inconsistent with local law:

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICA-

TION and FCtool "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR

IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-

INFRINGE-MENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Some states do not allow disclaimer of express or implied warranties in certain trans-

actions, therefore, this statement may not apply to you.

This information could include technical inaccuracies or typographical errors. Changes

are periodically made to the information herein; these changes will be incorporated in

new editions of the publication. IBM may make improvements and/or changes in the

product(s) and/or the program(s) described in this publication at any time without no-

tice.

Any references in this information to non-IBM websites are provided for convenience

only and do not in any manner serve as an endorsement of those websites. The mate-

rials at those websites are not part of the materials for this IBM product and use of

those websites is at your own risk.

IBM may use or distribute any of the information you supply in any way it believes ap-

propriate without incurring any obligation to you.

Any performance data contained herein was determined in a controlled environment.

Therefore, the results obtained in other operating environments may vary significantly.

Some measurements may have been made on development-level systems and there

is no guarantee that these measurements will be the same on generally available sys-

tems. Furthermore, some measurements may have been estimated through extrapo-

lation. Actual results may vary. Users of this document should verify the applicable

data for their specific environment.

Information concerning non-IBM products was obtained from the suppliers of those

products, their published announcements or other publicly available sources. IBM has



not tested those products and cannot confirm the accuracy of performance, compati-

bility or any other claims related to non-IBM products. Questions on the capabilities of

non-IBM products should be addressed to the suppliers of those products.

This information contains examples of data and reports used in daily business opera-

tions. To illustrate them as completely as possible, the examples include the names

of individuals, companies, brands, and products.

All of these names are fictitious and any similarity to the names and addresses used

by an actual business enterprise is entirely coincidental.

COPYRIGHT LICENSE:

This information contains sample application programs in source language, which il-

lustrate programming techniques on various operating platforms. You may copy, mod-

ify, and distribute these sample programs in any form without payment to IBM, for the

purposes of developing, using, marketing or distributing application programs con-

forming to the application programming interface for the operating platform for which

the sample programs are written. These examples have not been thoroughly tested

under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceabil-

ity, or function of these programs.

Trademarks

IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of Interna-

tional Business Machines Corporation in the United States, other countries, or both.

These and other IBM trademarked terms are marked on their first occurrence in this

information with the appropriate symbol (® or ™), indicating US registered or common

law trademarks owned by IBM at the time this information was published. Such trade-

marks may also be registered or common law trademarks in other countries. A current

list of IBM trademarks is available on the Web at http://www.ibm.com/legal/copy-

trade.shtml

The following terms are trademarks of the International Business Machines Corpora-

tion in the United States, other countries, or both:

FlashCopy®, FlashSystem™, IBM®, IBM FlashSystem™, Storwize®, System Storage®

The following terms are trademarks of other companies:

SAP, and the SAP logo, HANA are trademarks of the SAP AG, Germany, other countries,

or both.

Other company, product, or service names may be trademarks or service marks of oth-

ers.

IBM FlashCopy backup & copy solution for SAP …...HANA SnapShot, see SAP notes 2096000, 2039883....

Documents

Transcript of IBM FlashCopy backup & copy solution for SAP …...HANA SnapShot, see SAP notes 2096000, 2039883....