BMC - BMC Database Administration Solution for DB2 zOS WP000090DBS

White Paper www.butlergroup.com

1 Analysis without compromise

BBMMCC DDaattaabbaassee AAddmmiinniissttrraattiioonnSSoolluuttiioonn ffoorr DDBB22 zz//OOSS

A Return on Investment Analysis


2 BMC – BMC Database Administration Solution for DB2 z/OS © Butler Direct Limited

Written by: Martin Gandar

Published October 2008 © Butler Direct Limited

All rights reserved. This publication, or any part of it, may not be reproduced or adapted, by any method whatsoever, without prior written Butler Direct Limited consent.

INTRODUCTION

Methodology

BMC Software is one of the leading players in the market for DB2 database administration solutions with a portfolio of products (BMC Database Administration for DB2) that help optimise the effective use of DB2 on the mainframe.

To better understand how BMC Database Administration for DB2 provides business value to those that make it a key part of their mainframe systems management strategy, Butler Group undertook interviews with a number of users of these products. These interviews covered companies in a wide range of industries including medical, local government (including support for a criminal justice service), banking, insurance, and education. We supported this research with metrics gathered by BMC from an extended set of their customers and correlated this with our own technology assessments of BMC Database Administration for DB2 and its competitors.

THE BUSINESS PROBLEM

We contacted some of the largest mainframe environments in the World and some that were quite modest. One had 47 zIIP processors and used data sharing extensively and managed separate environments for development, testing, systems support and multiple production environments. Another had only one production database although this does contain billions of rows of data. The use of DB2 was obviously widespread and fundamental to the businesses that these environments enabled. In the case of the medical support group, the availability and reliability of these systems could quite literally be a ‘matter of life or death’.

For all of the sites interviewed, and we suspect any similar sites, non-availability of service is not acceptable and therefore they use a number of disaster recovery, mirroring, and backup solutions to ensure no loss of service to the running production systems. The use of BMC MAINVIEW for DB2 to automate the management and monitoring of production environments was widespread and tools such as BMC APPTUNE for DB2 were used to optimise transactions across them. However in a world where business agility requires constant change and updates to database structures (schemas), there is a need to provide a safe, reliable, and effective environment to manage these changes.

Putting changes into production involves the developers specifying or requesting changes to database structures which are then implemented by Database Administrations (DBA) staff initially in the development environment. The developers create new code and amend existing code to make use of the new structures. Putting this amended code into production requires a very well managed set of test processes that includes extensive regression tests against example production and historic data. Replicating changes made in the development schemas into the production environment is a highly complex and potentially dangerous business. Data needs to be unloaded and often reorganised before reloading into newly changed databases. Moving tested code and databases from a test environment through to production requires a planned period of outage that all sites are keen to minimise. To do this, the processes and supporting utilities involved to manage the DB2 environment need to exhibit the following profile:

www.butlergroup.com White Paper

© Butler Direct Limited BMC – BMC Database Administration Solution for DB2 z/OS 3

• High reliability – No failures or errors should occur during the process. If they do occur, it should be possible to restart processes quickly and easily or roll back to a previous stable state.

• High speed – Because outages of major production systems cost money and damage business continuity, the window is normally as short as possible and set at the least critical time. It helps greatly, however, if the processes run as fast as possible at the least possible cost.

• High degree of automation – Schema management tasks are complex. DBAs should only need to provide a high-level specification of the change required. Underlying processes required to implement the changes should be automated.

We should also note that the rate of growth of both data and activity in these production systems is extraordinary, whilst the increase in personnel offering support is at best constant. Training new staff in the use of IBM’s basic utilities and undertaking the complex tasks involved is increasingly rare with most of those we discussed it with saying that it was ‘unthinkable’. But before looking in more detail at the value these sites have gained from using BMC’s tools and utilities, let us quickly explain what tools and utilities are included in this solution.

THE BMC SOLUTION – DATABASE ADMINISTRATION FOR DB2

BMC Database Administration for DB2

BMC provides a set of tools and utilities that are designed to automate the complex processes involved in updating and managing DB2 environments with the aim of being easy to use, and fast and effective in their execution. BMC Database Administration for DB2 comprises the following tools and utilities:

• BMC CHANGE MANAGER for DB2 – automates the change process by analysing the request and determining the required changes, changing the structures, moving the data, and tracking the changes of all the DB2 environments.

• BMC CATALOG MANAGER for DB2 – automates the tedious, error-prone tasks of creating and dropping objects, maintaining user privileges, generating DB2 utility Job Control Language (JCL), and executing DB2 commands. BMC CATALOG MANAGER for DB2 simplifies the complexity of these required DBA activities and reduces the learning curve for less experienced DBAs.

• BMC UNLOAD PLUS for DB2 – unloads DB2 tables in the fastest time possible. It reads DB2 data sets (or table spaces) directly to obtain records, thus operating outside of DB2 and not competing for DB2 resources.

• BMC LOADPLUS for DB2 – loads DB2 tables (and subsets of data), enables conversion of data to a different format, sorts input data, dynamically allocates work files and image copy data sets, rebuilds and reorganises indexes, and collects DB2 object statistics during the load process.

• BMC COPY PLUS for DB2 – copies DB2 data and helps ensure recoverability.

• BMC SNAPSHOT UPGRADE FEATURE – is available stand-alone or as part of other solutions. It is a technology that allows concurrent application access during utility processing.

It is not our aim to describe these utilities in detail or to present a technical assessment of them; more detailed descriptions are available elsewhere. However, before reviewing the feedback we received from BMC’s customers, we should say a few words about the solution in general.

The solution has been designed to optimise the management of the DB2 environment by automating complex activities and reducing the complexity of the tasks that need to be undertaken by the DBA and Systems Administration staff. Considerable effort and constant development have also optimised the performance of the utilities such that they reduce the CPU load and other demands on other technical resources that need to be allocated to these tasks. The result should be that the tools more than pay their way, offering a return on investment profile that makes them a valuable addition to the support team’s armoury.



Customer Experiences

We started our appraisal by asking the customers for their general impression of the benefits afforded through use of BMC Database Administration for DB2 in the following areas:

• Increased database availability – Do you believe that these products are valuable in adding agility to the organisation and enabling more rapid changes that help the business?

• Improved performance of utilities – How does the performance of BMC tools and utilities compare with IBM and other rivals products?

• Increased productivity – How valuable an asset are these tools when considering the time and effort your staff spend on DB2 administration?

• Risk Mitigation – Are they a valuable asset in the fight to reduce costly errors?

Figure 1: BMC Customers’ General Assessment of the Quality of BMC Database Administration for DB2

The clients were asked to rate this on a scale of one (little value) to five (high value). The feedback was uniformly positive, averaging over 4.3 in all cases. Productivity was marked down a little in some cases on the basis that clients see room for enhancements or where they seldom made use of a function and had less knowledge to support a valuation. For example The DBA Team Leader at a major UK bank said “I dropped to four on the performance because there is always room for improvement, and many others things also contribute. As we don’t use LOBs, I made that a three.”

We then refined the questions to ask which of the facilities and capabilities each specific site found to be of greatest value. It is important to recognise here that this is not a judgement of the ability of the solution in each area, but of the value that each site gets from its use. If a site did not use the Table



Editor or need to support Large Objects (LOBs), for example, they may not have rated this feature highly, although others may have found it incredibly powerful.

Figure 2: Customers’ Valuation of Functionality Specific to Their Own Particular Site

Each of the eight clients interviewed could allocate up to five points for each of the capabilities reviewed, making a maximum of 40 points for each capability.

In examining the results, we found some very interesting evidence of the value of the solution.

The clients were unanimous in their regard for BMC CHANGE MANAGER used in association with other tools, such as BMC CATALOG MANAGER, to provide the ability to support changes that enabled business agility.

The sheer performance improvements offered by the solution over the basic IBM utilities was also highly complimented by all clients, and we shall see in the next section that there was strong evidence to support this view.

Reducing the likelihood of error in undertaking complex tasks and enabling less experienced staff to undertake these tasks was awarded high marks by all clients.

Enabling fast recovery from errors and failures (often that are nothing to do with the use of BMC Database Administration for DB2) was given high scores. The security offered by simplified and automated roll backs being particularly valuable.

Taking these figures and averaging them provides the summarised view of site-specific value given below. We stress again that this is a chart that shows the relative value of the features to these sites rather than an analyst’s technical evaluation of their particular capabilities. However, we would add that it is still an impressive result.



Figure 3: The Average Value Provided to Each Specific Site of Key Functionality

Customer and Butler Group Comments on Quality of the DB2 Administration Solution and BMC’s Support

Before we go further in the discussion of return on investment, it is worth mentioning the issue of quality.

BMC clearly has delivered a database administration solution that has the respect of its user base. The solution was generally said to be ‘up to BMC standards’ and that is clearly regarded as a hard standard to match.

Pete Gehlhoff of Milwaukee County was typical of the customers stating that “We have had phenomenal support from BMC”, the background to which we explain later in this paper. Generally the view was that BMC takes pride in the quality of products and support they offer. Getting their tools and utilities up to date with the latest releases of DB2 as quickly as possible and continuing to make their installation and use as easy as they can are some of BMC’s main goals. Support was consistent over many years with stable support teams of staff who where familiar with the sites they assisted.

Customer Comments on the Value and Capabilities of BMC Database Administration for DB2

BMC’s clients proved to be quite voluble on both the general and specific value they saw from the use of BMC Database Administration for DB2. They also talked about using those tools in conjunction with other BMC products. We will concentrate on the main product set in review here but will also mention other products as they seem relevant.



Availability Delivering very high levels of production availability is of prime importance to all BMC’s clients. Reducing the time to upgrade production environments and eliminating outages, planned or otherwise, is a constant concern. When problems occur during an upgrade it is vital that they can be fixed or reversed quickly with minimal impact on the production environment.

The DBA Team Leader of a UK bank said that “BMC CHANGE MANAGER for DB2 is the only product they have ever seen that allows you to back out any change from any point of failure without any manual intervention whatsoever. This instils great confidence when implementing any changes into DB2. The process is very slick and it’s merely a process of handing over three batch jobs. No matter how simple or complex the changes are they can be managed and backed out as needed.”

Pete Gehlhoff of Milwaukee County used an associated BMC product to help when errors occurred. He said “We do get errors we need to back out, for example when bad data gets through into production somehow and we use BMC Log Master for DB2 to sort it out.” They use BMC Log Master to undo SQL, and Pete said “It was really simple and worked really well. Just identify the SQL you want to undo and it works out how to do it most effectively.” He explained that by using BMC Log Master in conjunction with BMC Database Administration for DB2 he had been able to move physical locations of his mainframe without difficulty. Milwaukee County had determined that without such tools they would need a 24-hour outage to make the move. But Pete was able to negotiate two ten-hour outages separated over a period of days so that interim changes after the physical move could be logged and re-applied to the main database without an extended outage period. An associated point concerns the level of support offered by BMC during this move: Pete said that although he’d never used it before, “We had phenomenal support from BMC in helping to set this up. BMC’s Support team worked with us to optimise the rollover process.”

Todd Blandford at Duke Medical explained that improved resource optimisation had an add-on effect of delaying hardware upgrades and thus delivering greater availability of service. He said “We try to spend as much as possible of our budget on supporting the medical side of Duke. If we can delay an upgrade to the mainframe environment by a few months by optimising our resources using tools like BMC’s, then that puts the money where it’s most beneficial to the patients.”

Performance The senior DBA at a leading US financial house has one of the largest sites in the world with 120 DB2 subsystems and extensive data sharing. Some databases have billions of rows. He has 11 DB2 DBAs and 50 plus other DBAs in total He said “We see 50% CPU improvement through the use of the BMC Utilities in relation to IBM’s.” His site has some 1,500 BMC LOADPLUS batch jobs, thousands of BMC REORG PLUS and BMC COPY PLUS jobs. The financial house had tried to run without BMC’s tools but the senior DBA said “IBM often couldn’t finish the job when running reorganisations on very large (billion row) objects, usually because they could not get the resources required to complete the task.” He also said “You notice the difference most when multiple jobs are running together.” He uses BMC COPY PLUS to take weekly image copies as they don’t keep their logs for very long. When doing so he found the wildcard feature useful in the allocation of Generation Data Group (GDG) files and, as he has thousands of these, the performance improvements are crucial. His most telling comment was “We take the BMC utilities for granted because they have just become a part of our life.”

The Operations Director at a US banking group saw even greater performance improvements and says that BMC LOADPLUS and UNLOAD PLUS are three to four times faster than IBM. He runs 3,000 nightly batch jobs and states that “We couldn’t run the workload using the standard IBM utilities.”

The DBA Team Leader at the UK bank we interviewed uses BMC LOADPLUS and BMC UNLOAD PLUS extensively where data is converted for loading. He says “BMC LOADPLUS is used for performance and to avoid writing programs for data conversion which is automated in BMC LOADPLUS. We can load and convert without new coding, which saves huge amounts of manpower.” He said “We are making a significant investment in getting the best out of these utilities.”

The UK bank had run a series of tests repeatedly using the same data and test environments for the IBM utilities against the BMC Database Administration for DB2 alternatives. The results showed improvements in CPU utilisation of BMC UNLOAD PLUS and BMC LOADPLUS of over five times that of the IBM equivalents with associated products like BMCSTATS and ONLINE REORG being over three times more efficient. In all



cases, the amount of workspace used for sorting was considerably reduced offering a further saving of resources.

Todd Blandford of Duke Medical confirmed a raw improvement of performance at 20% to 40% up on IBM utilities whilst Jean Bose of Newell Rubbermaid said they were at least 15% to 20% faster.

Butler Group would suggest that whichever experience you regard as the benchmark, these performance improvements are going to help increase availability of production systems and reduce resource usage and therefore the cost of maintaining them.

Productivity If BMC Database Administration for DB2 was well regarded for performance, then the applause was deafening when it came to productivity improvements. There were a number of themes to this:

• Ease of use and adoption.

• As an enabling tool that allowed staff to undertake complex tasks that would otherwise be too important for them to be allowed to undertake.

• As a means of automating repetitive tasks.

Ease of Use

BMC CATALOG MANAGER scored highly on the issue of ease of use and support for those less skilled, with Pete Gehlhoff saying that “I have thought of running a half day’s training in BMC CATALOG MANAGER to help others outside the systems department see things for themselves rather than come to me with questions.” This is something the Systems Manager at a major US insurer has put into practice. He said “BMC CATALOG MANAGER is user friendly, very Intuitive and straightforward to use. We have a two-hour class and then they are off and running.”

Jean Bose, Senior DBA at Newell Rubbermaid said “BMC CATALOG MANAGER is very accessible and easy to learn. It’s good for developers particularly the menu-driven nature.”

The US bank’s Operations Director said that “The ease of use of BMC CATALOG MANAGER is key when looking into the DB2 catalogue and navigating around from the index to the packages without having to know a lot of other details. It’s another level of skill they just don’t need to have. The comparison between environments is great when moving between production, test, and development [he has multiple test environments]. You can say, I want to migrate changes one, two and five but not three and four, for example.” The Operations Director also liked the functionality saying “You can make sure things don’t get missed using the compare function, and this helps out greatly.”

The functionality was a point picked up by the senior DBA at the US financial house who said “There are features in BMC CATALOG MANAGER V9 that haven’t made it to the IBM admin tool yet.”

An Enabler

BMC CHANGE MANAGER is clearly a powerful ally of the busy DBA as it reduces the effort by hiding or simplifying the complexity of tasks and automating activities that would otherwise be a severe burden. Pete Gehlhoff said “We don’t have BMC in the test Logical Partition (LPAR). It’s kind of shocking to work there. You don’t realise how much you depend on the tools sometimes.” We asked him to give an example, and he said “Trying to recreate views efficiently became a major issue for us, as they are defined in a long text string that you have to break up into 80 byte data sets. I had to do it without the BMC tools in the test environment but using CHANGE MANAGER would have made making the views and grants simple. We also use CSID ISPF Editor to get view text into something that could be made into some SQL to create the Data Definition Language (DDL). A half-hour activity would take many hours, something like a ten- to twenty-fold increase in effort.”

The Systems Manager at a US insurance company said “BMC CHANGE MANAGER adds productivity. Where we have multiple development environments feeding a single production environment using BMC CHANGE MANAGER, I am able to create migration profiles and push those changes out and keep things in synch fairly easily. This gives me time to work on other things and avoids potential problems. Without these tools and utilities there is a significantly high chance that we’d get out of synch. Doing it manually could take ages, something like five days rather than under a day as it does with BMC CHANGE MANAGER.”



Our US bank’s Operations Director said “I can’t imagine doing the changes without BMC CHANGE MANAGER. If we didn’t use it there would be a massive additional training burden. Simpler changes involving setting up the loads and unloads and manipulating the job cards and objects would probably be twice the time. Making complex changes with BMC CHANGE MANAGER takes 25% of the effort that would be required to do it manually.”

He also pointed out the benefits of BMC CHANGE MANAGER in its ease of use and transparency. “We have a lot of DBAs supporting on-call rotation. A major benefit of the BMC tools is the consistency and ease of passing workloads between staff. People don’t need to know all the in-depth details. They just tell BMC CHANGE MANAGER to make this table look like that and let BMC CHANGE MANAGER organise and execute the work. It definitely helps folks get up to speed a lot faster to fulfil a DBA function.”

The bank has 40 DBAs in total; some have more focus on SQL tuning while others are most concerned with database changes. The BMC tools and utilities enable flexibility of work allocation and allow new DBAs to make table changes that they might otherwise not be allowed to undertake, letting their senior DBAs work on other things rather than routine activities. The Operations Director said that “Without these tools we would have to have more staff that have experience and greater depth.”

The UK bank’s DBA said “If we didn’t have BMC CHANGE MANAGER, we would need a lot more DBAs – it would just be so hard. I’d probably need to add two more to the five we have already. Things you run now in minutes would take a day or two. To manually work out object dependencies and how to unload, convert, and load data would be a slow, laborious process.” The DBA had previous experience at another site where it took him two months to work out how to implement a change and undertake it and found out later that the task took him an hour with an early version of BMC CHANGE MANAGER. The DBA also said that “In two hours I can take all definitions from three production subsystems and merge them into a single system test environment. I don’t have a clue how long that would take to do manually because we are talking thousands of objects. Many weeks at least!”

Todd Blandford of Duke Medical relies heavily on BMC CHANGE MANAGER to control his main production subsystem and separate Quality Assurance (QA), development, and test environments. He said “We save a quarter to a half of an FTE (Full-Time Employee) due to its use. We have a sandbox environment in which the developers make changes and check these with us, requesting database changes as required. Changes may be minimal or involve multiple changes like ten to 20 table changes that result in the need to rebuild the database; for example, when a new tab on a Windows-based application screen created new application functionality that impacted our database objects. We use BMC CHANGE MANAGER to evaluate the impact and to manage the changes. The developers may use their own names and the Systems Department must apply the standards consistently (replacing names, etc.) to get the code and JCL compliant with standards. We then have to propagate all the required changes. BMC CHANGE MANAGER builds the DDL unloads, loads, and so on, and moves the data to a new subsystem. When they get to the next stage they take that script and run it again to move to the next stage.”

The time saving when changing ten to 15 tables, each with multiple aliases and indexes, is considerable. He says that “What would have been half a week’s time is now accomplished in half an hour’s time (and that repeats for each stage of moving data through the environments). The first time is the most time-consuming and the following migrations are less effort, but having software that is intuitive and panel-driven automates much of the process and is very valuable.”

Pete Gehlhoff of Milwaukee County cites BMC COPY PLUS as being an invaluable productivity aid saying that “Without it to make Image copies we would need to change to IBM. Not a great handicap but we would need to learn new tools that are not as effective.”

The US bank uses BMC COPY PLUS to make backups. The Operations Director uses image copies to roll back changes and has disk mirroring for disaster recovery. He says “It gets to the point where you can’t imagine not having the tool. When it would take you four times as long – big savings all round there.”

The Value of Automation

There was also general agreement on the value of automation and the productivity delivered across the whole product set.

Jean Bose of Newell Rubbermaid said “Products such as BMC LOADPLUS and BMC UNLOAD PLUS save CPU time, but the Admin products generally save more DBA time.” She had two DBAs working on DB2



management and without these tools she said “I’d have needed another full-time person and a part-time one to cope with busy periods.”

Jean said that they have needed to use the tools to clone a new environment when taking on a new client. “It took less than 60 days from corporate purchase to have moved over all our data rather than the months it might have taken manually.” She used BMC CHANGE MANAGER with each different division to set up a separate set of near identical databases. She used BMC UNLOAD PLUS and BMC LOADPLUS on around 400 tables in each of 17 divisions on three instances to move the data. This was “very efficient and saved an enormous amount of time.”

The US insurance company’s Systems Manager said, “Without these tools, everything would be manual. It takes nine to 12 months to make a system programmer productive without these tools but maybe 30 days using them. Doing things manually may help you learn but getting people productive earlier is more important. Automation is necessary to support the call for ever greater productivity. We haven’t increased the number of people since 2004 although the workload has doubled. I could attribute the cost of a couple of staff on the systems management team to this saving, and it has alleviated the need to hire additional DBAs with all the associated cost.”

Todd Blandford of Duke Medical said that although raw performance was useful he really saved the time of people sitting in front of a terminal. “With BMC building things automatically, we see dramatic overall time savings. Things that used to take a day or an hour come down to between ten minutes and half an hour. I would have to have two other people full-time as a minimum to maintain the business if we did not have these tools.” Todd also made an interesting point regarding recruitment saying that “It saves time on staff turnover and training. We look for people with BMC background and they can easily adapt to the new environment with virtually no training, just different data. Fortunately BMC has enough of a footprint that we can always do this.”

Risk Mitigation In all the clients’ sites we questioned, we asked how fundamental the DB2 environment was in supporting their core business. They were unanimous in saying that they just cannot accept outages and need tools and utilities like BMC Database Administration for DB2 to help ensure that it doesn’t happen or that they can rapidly recover if the worst should occur.

The US bank has one major production environment with all the data in a single DB2 environment. This is a large data-sharing environment between nine environments, and one additional group that is inactive as a support environment. Work can run on any one of the eight active members. The Operations Director says “All eggs in one basket is good from an administrative standpoint; but if something goes wrong with DB2 it would take everyone out at the same time!”

In such an environment rigour is essential so that things don’t get missed, “such as objects like triggers in the database and things you don’t use so often. BMC CHANGE MANAGER looks after them, whereas a DBA might miss them. Sorting out things you missed later on is much more effort. The cleanup can be a lot more time than the change took you to make in the first place, and there could be other consequences along the way.”

Where the bank uses PeopleSoft tools to make changes, things do get missed; such as views not being created correctly. The Operations Director said “Sometimes we have to go through and clean them up. Larger changes will often need some significant cleanup with BMC tools. It’s not so much a cleanup really, but an impact analysis asking questions like ‘what did we send out to customers?’ If there was a problem we then have to put back an object to a previous state which is not so hard, but it’s a matter of what’s happened between the error and correcting it. Using the BMC tools to manage changes significantly reduces this risk.”

The UK bank’s DBA said he did occasionally have outages, but they are related to DB2 bugs or hardware. “The BMC tools have never caused a problem that has taken DB2 out.” BMC CHANGE MANAGER helps them to compress and reorganise if a partitioned table space is filling up. They use BMC DASD MANAGER (an associated BMC performance tool) to predict such space growth and BMC MAINVIEW to monitor and initiate other corrective actions automatically.

The Senior DBA of a US Services company uses BMC CHANGE MANAGER to ensure error-free migrations. He explained “To change ten tables, add two, and remove two old ones, we would specify the change in the test environment and then replicate the work list for system test to compare with the schema and examine the scope rules in the schema. Scope rules explain what should or shouldn’t be in the new environment. Change Definition Language [BMC’s proprietary extension] defines changes so that you can


© Butler Direct Limited BMC – BMC Database Administration Solution for DB2 z/OS11

say; ‘add this before this column’, for example. We are then able to turn out a very repeatable error-free operation to do our migration. Without this tool, it gets very complex to unload, reload, and find the packages required and to check for Referential Integrity. I’m a senior DBA, but I wouldn’t like to have to do that. It would be a day-long process to migrate a database manually with changes. The BMC tools can take this down to minutes and I can trust a non-senior DBA to do that.”

Our Systems Manager at the US insurance company said “Sure, it removes issues. It allows you to exactly replicate what you have done in the test environment and put it into production. What you have tested is what you get! So you stop people changing things that get other things out of synch and that can then cause an outage. It has probably prevented lots of outages.”

He further explained that a loss of production was a costly issue: “In our business there are Independent Agents as well as Direct and Captured Agents who purchase insurance premiums on behalf of their customers. Application outages cause us the loss of lots and lots of premium income. Independent Agents would just choose an alternative insurer if we can’t respond to them instantly, as they go for multiple lines of insurance in multiple companies. If my Web site isn’t available they choose someone else. My database, my system, and I have to be up and available at all times.”

We asked him to try to put a value on that, and he suggested “I can lose thousands and thousands of dollars every minute I am down.”

Todd Blandford of Duke Medical cannot afford outages. His DB2-based Clinical Data repository (CDR) is critical to the business. “It’s critical because many of the applications areas supporting the clinicians, e.g., in emergency, labs, oncology, or radiotherapy are separately run by department heads. The doctors need to pull together every bit of information on a patient from all the systems that gather it. To do this, they have created a single database of information: one table with over two billion rows of data on every patient and their status throughout their history. We cannot afford to lose access to that data. We deal with a different currency. This is critical information for life–and-death decision making, so speed and accuracy and 100% availability are vital.”

Duke Medical like and use BMC RECOVERY MANAGER (a complementary tool) to manage an off-site hot site recovery mechanism. Todd said “It was impossible before we had these tools to do a test of whether the recovery mechanism worked because the hot site couldn’t recover within a period of 48 hours.” It took them too long to work out what had to be recovered and create the manual scripts. BMC RECOVERY MANAGER smoothed out the manual processes and generates the required JCL. Now they just submit a job with the recovery JCL that recovers the data in 10-16 hours.

DATABASE ADMINISTRATION FOR DB2 RETURN ON INVESTMENT

The customer experience presented in the previous section provides strong evidence of the potential value offered by BMC Database Administration for DB2. Turning this into a potential financial saving is of course difficult because there are major differences in the complexity, scale, and working practices across the differing implementations. However if we take very cautious figures for what has been achieved, and are careful not to overstate the findings, we can develop a reasonable and independent model based on their experience showing the potential savings made though the use of these tools.

Performance Savings All of the clients we interviewed gave glowing reports of the performance improvements delivered by the use of BMC’s utilities versus those provided by IBM. The savings quoted varied considerably, with some saying a 20% to 40% improvement in CPU savings to others stating that the tools were four or five times (80%) faster. Let us take conservative figures of around 40% saving in CPU and elapsed time as our basis. Now consider a medium-sized financial institution running a combination of BMC LOADPLUS and BMC UNLOAD PLUS jobs totalling 500 in any one week on large tables, some of them approaching 500 million rows. If we use conservative industry figures for the cost of elapsed time and CPU and say the tables contain 50 million rows we could express potential savings in the following equation:

Cost of elapsed time = US$1.75 per second

Cost of CPU = US$1.5 per second



Number of loads and unloads per year = 500*50 = 25,000

If a load would take 40 seconds using IBM’s utilities and 30 seconds CPU then the potential savings using BMC’s utilities and tools would be 16 seconds on elapsed time and 12 on CPU per job. (We note here that BMC state that the savings are greater with jobs that run for minutes or hours but we do not wish to overstate these savings and so are using very conservative figures.)

Elapsed time saving = (Elapsed time saved per job * Elapsed cost per second) * Number of jobs

CPU saving = (CPU time saved per job * Elapsed cost per second) * Number of jobs

Elapsed time saving = (16*1.75)*25,000 = US$700,000

CPU saving = (12*1.5)*25,000 = US$450,000

Total saving = US$1,150,000

Productivity Savings First of all we’ll look at some typical specific examples of improved productivity, then we’ll consider the more general comments on staffing stated by the clients.

It was clear from discussing productivity savings with these clients that the use of BMC CHANGE MANAGER was delivering really significant value in three areas: reducing risk (by eliminating potential errors), saving staff time, and providing the option to use less experienced staff to undertake changes that they would otherwise not be authorised to attempt.

The evidence presented by the US bank offered us potential savings of 80% when undertaking complex changes, taking a quarter of the time they would otherwise have taken manually. Let us say that we have ten environments with five major changes a year being undertaken on them. If we say that the manual changes took two staff four days (32 hours each) and are now reduced down to one day then the saving per change is 24*2 = 48 hours. Taking the cost of a trained DBA at US$100 per hour then the total cost saving here could be described as follows.

Value of improved staff effectiveness = (Cost of DBA per hour * Hours saved) * Number of changes per environment * Number of environments

Value of improved staff effectiveness = (100*48)*5*10 = US$240,000

The Senior DBA of the US Services company talked about the cost of database migration from one environment to another, with processes that might take days reducing to minutes. He talked about making selective changes involving additional tables and the dropping and adding of columns as he migrated databases across environments as part of his normal development and production systems update cycle. He did state that the changes got easier (or the savings less) as you moved through multiple environments, so we’ll take this into account on our estimate of savings.

Suppose the establishment of the required changes initially is reduced from two days down to two hours and that subsequent migrations are reduced from what would have been eight hours down to two hours and that we are talking about four additional environments in the chain. If such migrations occur 20 times a year and involve two staff then we get the following:

Migration savings = (Initial saving on first migration + (Number of subsequent migrations * Saving on subsequent migration)) * Number of migrations

Migration savings = ((Hours saved on first migration * DBA cost per hour * Number of staff involved) + (4 * Hours saved on secondary migration * DBA cost per hour * Number of staff involved)) * Number of migrations

Migration savings = ((14*100*2)+(4*6*100*2))*20 = US$152,000

Jean Bose of Newell Rubbermaid discussed the massive savings they made when loading and converting data into sets of cloned environments as they took on new clients. She mentioned the load of 400 tables in 17 divisions with three instances and the massive savings in time and effort saved completing the clone and load of a client in 60 days (or two months) rather than many months if undertaken without BMC’s tools. If we simply say that they halved the time and effort of each conversion using staff who cost US$200,000 per annum to employ and undertook only four such conversions per year then we could define the savings as follows:



Savings in the load and conversion of data = Time saved per conversion in months * Cost of conversion staff per month * Number of conversions per year

Savings in the load and conversion of data = 2*200,000/12*4 = US$133,333

We might also note that many of the benefits related to these migration tasks are reductions in the CPU costs of the utilities that are executed to move the data. BMC pointed out to us that BMC CHANGE MANAGER generally builds all the migrations automatically. There would in reality be a cost at sites not using their tools to build application programs to convert the data although we haven’t estimated that cost as it is difficult to quantify.

If we look at the comments made by all the clients on staffing, it is clear that they agree that they would need to employ additional DBAs, and probably need to undertake more training, if they were not able to use BMC’s tools and utilities yet still had to handle the ever increasing workload. A smaller company employing two or three specialist DBAs and Systems Staff would need one or two extra whilst a major site with 20 plus such staff might need another five at least.

There is also the added burden on training and the potential savings in costs through the use of less skilled staff being able to undertake tasks such as database changes that would otherwise have required more experienced staff. It’s difficult to put a blanket value on the likely training costs involved, but one client suggested that the tools enabled a new DBA to get up to speed in one month as opposed to between nine and 12. If we suggest that during the training period the DBA was half as productive as a fully trained DBA who costs $150,000 per year to employ and at any time we are training up two DBAs, then it is possible to put a value to this aspect:

Value of improved staff capability = Number of staff being trained * (Increased number of months in trainee status / 12) * Cost of DBA * Productivity reduction

Value of improved staff capability = 2*(8/12)*150,000*0.5 = US$100,000

However you might say that this rather misses the point. Without these tools used for training or otherwise there would be the need to employ additional staff. This could mean an increase in staff costs of anything from US$150,000 for one new DBA per annum and up to US$750,000 for an additional five DBAs.

The additional training required would add an additional burden to productivity and the time at which they became productive. So typically the staff cost/value associated with not having the BMC tools might be more like US$250,000 for a smaller site with one new experienced DBA and one trainee, and US$950,000 for a large site with five additional staff including two trainees. We don’t want to double-count or overstate this aspect but these figures do tend to support the general opinion that there would be significant additional staff costs without the use of BMC Database Administration for DB2.

Availability Putting a value on increased availability is always difficult. In our interviews we found a number of ways that BMC Database Administration for DB2 had contributed to increased availability and enhanced the effective capacity of their installations. But the clients hadn’t measured the value specifically, but it is not difficult to determine some rational analysis of the value of the increased availability that they imply.

Let’s start with an example that relates to enhanced capacity and cost effectiveness of the environment.

Duke Medical works hard to delay the need to upgrade their environment. They save in two areas: the reduced cost of financing the hardware and software upgrades for the period that they can delay the upgrade, and depreciation costs for equipment.

If we say that a typical upgrade cost of technical resources is around US$6 million and finance costs are running at 5% per year and that the upgrade is delayed by six months, then we can state the saving as follows:

Deferred cost of financing = (Typical upgrade of technical resource cost * Finance cost) * Time in years

Deferred cost of financing = (US$6 million*5%)*1/2 = US$150,000

Considering the depreciation cost over the year, the value of the additional facilities in the enhanced mainframe environment will typically decline by 20%; if we can avoid the need to install them we will not suffer this loss in value. This provides a potential additional saving through the decline in resource value of up to US$600,000 over a six-month period:



Depreciation cost = (Typical upgrade of technical resource cost * Annual depreciation rate) * Percentage of year delayed

Depreciation cost = (US$6 million*20%)*50% = US$600,000

All of the clients interviewed stated that there were productivity and performance improvements that enabled them to ‘get the job done’ more quickly and with greater agility than without the use of the BMC tools. What this results in must be greater availability and earlier implementation of upgrades to their production environments. It is easier to put a value on the potential damage that a loss of these environments would involve than the increased value of a better production environment, as this is very client-specific indeed. However Butler Group would like those reading this paper to consider the value to their organisations of the earlier introduction of improved services together with improved production availability and to consider its value in their own environment.

Despite the comments just made, we will try to present an example of the value that increased availability of the production systems through the elimination of downtime and more effective systems management gives to the business.

If the downtime due to planned maintenance is reduced, errors are fixed more rapidly, and improved capability is brought into production more smoothly, it is fair to suggest that the production systems will be performing more optimally in the delivery of the business applications they support. If we consider that the average improvement to productivity is merely 1% to a business making US$50 million profit then:

Value of additional availability = US$50 million*1/100 = US$500,000

Risk Mitigation BMC’s clients work hard to avoid outages, but should they occur they are very costly indeed.

The actual true cost of an outage could be in human life – as in Duke Medical – or in simpler financial terms – as in the US insurance company’s case. The loss of premium income is clearly simpler to put a value to and so we will use that as our example, although again we would invite you to consider the cost of an outage in your own environment and the contributing factors regarding your use of DB2 that could lead to it.

Let’s say that the number of outages per year had the site not used the full set of BMC tools would be ten and that BMC Database Administration for DB2 is a major factor in reducing these by 20%. If the average length of such an outage was 90 minutes and the cost in loss of income is US$5,000 per minute, then the cost of outages saved by the administration solution is as follows:

Severe outage saving = (Number of potential outages * Outage period in minutes * Cost per minute) * Percentage contribution of admin tools to their elimination

Severe outage saving = (10*90*US$5,000)*20% = US$900,000 per year

Other than outages, there are clearly further risks such as errors brought in through migration mistakes and reduced quality of data management that could lead to regulatory infringement, litigation, or simply the loss of business. In Duke Medical’s case it is easy to see how the failure to provide the appropriate summary of a patient’s care record could result in potential damage worth millions of dollars.

Butler Group Comment on ROI Values If we add up the various savings described above we come out with a total that just exceeds US$4 million. This is, we believe, quite a realistic figure for a significant organisation where the optimal use of the DB2 environment is likely to contribute significantly to the value of the business it supports. You may, however, wish to eliminate some aspects of the Return On Investment (ROI) calculation and vary the figures in other parts to match your own environment. We do believe that the figures above reflect the true value that the BMC Administration for DB2 solution delivered to the clients that we interviewed.

CONCLUSION

There is a very high regard for BMC’s tools generally, and the BMC Database Administration for DB2 tools are clearly what their clients call ‘up to BMC standards’. As our DBA Team Leader at the UK bank put it so



well “The BMC tools may be considered by management to be a Rolls Royce solution, but we use them and no-one can get close to them or match their performance. The products do what they say they can do on the tin, and that’s all you could ask for.”

Butler Group were very impressed indeed by the level of acclaim that BMC’s customers gave to these tools and the fact that they would find it hard to cope with their day-to-day workloads without their use.

The individual commendations and examples proved to us that the value of the tools was very significant and we have reflected this in the Return on Investment calculations that we believe to be a true indication of their worth.

Not all clients will gain US$4 million in reduced costs, reduced risk, and increased performance and productivity, but there is no doubt that these tools offer a strong potential ROI and should be seriously considered by anyone running DB2 in the mainframe environment.

Contact Details

Corporate Headquarters BMC Software 2101 CityWest Blvd. Houston Texas 77042 USA Tel: +1 (800) 841 2031 E-mail: [email protected] www.bmc.com

London Office Assurance House Vicarage Road Egham, Surrey TW20 9UY UK Tel: +44 (0)1784 478 000 www.bmc.com/uk

Headquarters: Australian Sales Office: End-user Sales Office (USA):

Shirethorn House, Butler Direct Pty Ltd., Butler Group, 37/43 Prospect Street, Level 46, Citigroup Building, 245 Fifth Avenue, 4th Floor, Kingston upon Hull, 2 Park Street, Sydney, New York, NY 10016, HU2 8PX, UK NSW, 2000, Australia USA

Tel: +44 (0)1482 586149 Tel: + 61 (02) 8705 6960 Tel: +1 212 652 5302 Fax: +44 (0)1482 323577 Fax: + 61 (02) 8705 6961 Fax: +1 212 202 4684

For more information on Butler Group’s Subscription Services please contact one of the local offices above.

Important Notice This report contains data and information up-to-date and correct to the best of our knowledge at the time of preparation. The data and information comes from a variety of sources outside our direct control, therefore Butler Direct Limited cannot give any guarantees relating to the content of this report. Ultimate responsibility for all interpretations of, and use of, data, information and commentary in this report remains with you. Butler Direct Limited will not be liable for any interpretations or decisions made by you.

*96934*

BMC - BMC Database Administration Solution for DB2 zOS WP000090DBS

Documents

Transcript of BMC - BMC Database Administration Solution for DB2 zOS WP000090DBS