The Story of a Week in Production on DR Site - Francois Vancoppenolle

1

The Group P&V is comprised of the brands P&V, VIVIUM, Actel, PNP, Euresa-Life and Arces

The story of a week in

production on DR site

[email protected]

N.R.B. Mainframe Day 2015

27 May 2015

mailto:[email protected]

2


Overview

• Introduction : P&V Group

• Mainframe architecture

• DRP Scenarios

• Testing the DRP Scenarios

• Test DRP Scenario on Production partitions

• Test Scenario DRP P&V 2012

• One week in DRP : Lessons learned

• What’s next ?

3


Introduction : P&V Group

Market Position in 2013 (in Belgium) :

• NON-Life : 6th (702.796 k€)

• Life : 7th (748.642 k€ )

4


Overview



• DRP Scenarios





• What’s next ?

5


Mainframe Architecture (1/4)

Bruxelles

Antwerpen

Herstal

Villers-le-Bouillet

6



Bruxelles

AntwerpenHerstal

Villers-le-Bouillet

119 Km

82 Km

32

Km

46

Km Network

configuration

7


EC12 EC12

DS8700 DS8700

VTS7700 VTS7700

PPRC

Ptp

Switch Switch

HerstalVillers-le

-Bouillet

8



P&V LPARS on NRB infrastructure :

ZOS : ZVM :

LPAR PROD 1

LPAR PROD 2

LPAR QLTY 1

LPAR QLTY 2

LPAR TEST 1

LPAR TEST 2

LPAR SYSTEM

LPAR PRODUCTION :

- 13 Linux Production

- 15 Linux Quality

- 15 Linux Test

- 2 Linux System

LPAR SYSTEM

- 2 Linux System

9


Overview



• DRP Scenarios





• What’s next ?

10


DRP Scenarios (1/6)

4 Scenarios are taken in consideration

• Villers-le-Bouillet Down

• Disks and/or VTS7700 Down at Herstal

• Z13 Herstal Down

• Herstal Down

11


EC12 EC12

DS8700 DS8700

VTS7700 VTS7700

Switch Switch

HerstalVillers-le

-Bouillet

12


EC12 EC12

DS8700 DS8700

VTS7700 VTS7700

Switch Switch

HerstalVillers-le

-Bouillet

13


EC12 EC12

DS8700 DS8700

VTS7700 VTS7700

PPRC

Ptp

Switch Switch

HerstalVillers-le

-Bouillet

14


EC12 EC12

DS8700 DS8700

VTS7700 VTS7700

Switch Switch

HerstalVillers-le

-Bouillet

15


EC12

DS8700

Herstal Villers-le

-Bouillet

EC12

DS8700

DRP Scenarios (6/6)

4 situations

to test(VTS is not an issue)

16


Overview



• DRP Scenarios





• What’s next ?

17


Testing the DRP Scenarios (1/3)

Every 6 months, the 4 scenarios are tested on :

• The 2 ZOS QLTY Partitions + ZOS System Partition

- Check that system engineers can logon in TSO;

- Check the sysplex between the 2 partitions;

- Check the connexions are correct (VTS/DS8000/OSA/…)

• The ZVM System Partition

- Check that system engineers can logon in CMS;

- Check that system engineers can logon in Linux;

- Check the connexions are correct (DS8000/OSA/…)

18



IF SYSTEM ENGINEERS CAN PERFORM

-> A LOGON IN 1 ZOS/TSO,

-> A LOGON IN 1 ZVM/CMS,

-> A LOGON IN 1 LINUX

THEY ARE ABLE TO SOLVE ALL

PROBLEMS ON OTHER PARTITIONS.

19



WHAT DOES THIS TEST PROVE :

Minimum cabling is available

System Engineers should be able to

restart all partitions and to solve all

problems.

WHAT DOESN’T THIS TEST PROVE

All partitions will start without problem

Everything will work as it should in DRP

20


Overview



• DRP Scenarios





• What’s next ?

21


Test DRP Scenario on Production Partitions (1/2)

To be sure that the production partition will have no

(?) problem in DRP, you have to test this situation…

and test EVERYTHING.

But this is impossible…

One example: will you during the test execute the

batch programs that runs once a year ?

Test as much as possible

test the most important things !!!!

22


What to test ?

• All starting taskes and Specific programs

(Scheduler, Tape manager, Output managers, …)

• Databases;

• Transactional programs (CICS, IDMS, …);

• The Batch process.

• Connectivity available and sufficient.

• Power of the CPU sufficient.

• “all other things that you don’t think about…”

Test DRP Scenario on Production Partitions (2/2)

23


Overview



• DRP Scenarios





• What’s next ?

24


Test Scenario DRP P&V 2012 (1/2)

In October 2012, upon proposal of the mainframe

system engineers, P&V has decided to run one week

in DRP mode.

Purpose :

• Testing the batch process seriously.

• Check that connectivity is sufficient.

• (Check that the power of the CPU is sufficient).

• “Test all other things that you don’t thinks about…”

25


EC12

DS8700 DS8700

VTS7700 VTS7700

Switch Switch

HerstalVillers-le

-Bouillet

PPRC

Ptp

26


Overview



• DRP Scenarios





• What’s next ?

27


One week in DRP : Lessons Learned (1/5)

WHAT WENT WELL ?

• “Yes, we can” : We are on the cloud but …

everything was performed by ourself without the

intervention of NRB (except CBU activation).

• It should be possible to switch in DRP mode in 1

hour but 3 additional hours needed for collecting

the licence keys for the other CPU.

Trends since 2012 : DRP key available on the web.

• “No problem to startup : ZOS, ZVM and Linux.

28



WHAT FAILED ?

• Cabling was not sufficient for the I/O after the W.E.

Cables added by NRB on Monday => Problem solved.

Switch installed by NRB 3 months later

• Cables where added by NRB, but we (P&V) fail to

activate all of them (detected on Thursday)

Avoid defining new cables during DRP. With the switch

installed by NRB, this should not be necessary anymore.

29



WHAT FAILED ?

• HMC consoles not reachable on DRP site.

IPL performed from the consoles at Herstal;

Changes performed by NRB to make the HMC console

available.

30



WHAT FAILED ?

• Batch was OK except :

Bugs in some “system” programs based on the LPAR

name => Programs have been fixed.

Problem detected with the BACKUP that uses DISK on

one DS8700 and that are not defined on the other

DS8700 => Temporary solution (for the DRP test) : Disk

added on the other DS8700; Definitive solution : Disk

space added on the other DS8700 & Flash Copy S.E.

implemented.

31



WHAT WAS NOT PROVEN WITH THIS DRP TEST

• CPU, memory and cabling are sufficient if ALL

partitions (all NRB customers) work on the DRP

infrastructure.

To prove it, all customers should perform the same test

at the same time. Not easy to plan…

32


Overview



• DRP Scenarios





• What’s next ?

33


What’s next ? (1/2)

• All problems detected during the test are solved

(we hope so), but another test need to be planned.

Next DRP test : October 2015

34


What’s next ? (2/2)

• Scenario not yet tested :

Suppose that the disks/tapes are destroyed partially or

totally on the primary and on the secondary site (because

of an “error” of a Storage Administrator/System Engineer)

In 2016 (? - when NRB will replace the disk infrastructure),

we will restore a tape backup on it and try to restart the

partitions on the restored disks…

35


The Story of a Week in Production on DR Site - Francois Vancoppenolle

Technology

Transcript of The Story of a Week in Production on DR Site - Francois Vancoppenolle