TECHNICAL NOTE Sender: Recipient: Attached Files: Number ......2.4.4 Metric Data Tablespaces Format...

$: TECHNICAL NOTE Sender: Recipient: Attached Files: Number ......2.4.4 Metric Data Tablespaces Format C{Chnl}_{Type}_{AggSet}_{TSpace} with: {Chnl} The channel set number (from 1 to$
Page size suitable for… A4

(European) Letter (U.S.)

TECHNICAL NOTE

Recipient: Number of Pages: Sender: Attached Files:

Technical Publications Internal None 24 External

Netcool/Proviso Backup Procedures –Technical Note

.............................................................................................................................4 1 DOCUMENT OVERVIEW

..................................................................................................................4 2 BACKING UP THE DATABASE

..................................................................................................................................................4 2.1 Warnings

...................................................................................................................................4 2.2 Reference Material

...................................................................................................................................................5 2.3 Concepts ............................................................................................5 2.3.1 Direct Loading / UNRECOVERABLE

...................................................................................................5 2.4 Terminology and Naming Conventions .......................................................................................................................5 2.4.1 Metric and Meta Data ................................................................................................................6 2.4.2 DataChannel Components ...................................................................................................................6 2.4.3 Channel Table Set Type ..................................................................................................................6 2.4.4 Metric Data Tablespaces ....................................................................................................7 2.4.5 Metric Data Tablespace Datafiles

......................................................................................................................7 2.5 READ ONLY Tablespaces

.............................................................................................................................................7 2.6 Hot Backups

...............................................................................................8 2.7 Netcool/Proviso Tablespace Management

........................................................................................................8 2.8 Oracle Recovery Manager (RMAN)

.............................................................................................................9 2.9 Data Channel Loading Summary

............................................................................................................................10 2.10 Data Channel Reloads

.......................................................................................11 3 NON-METRIC TABLESPACE MANAGEMENT

......................................................................................................................................11 3.1 Backup Method

4 METRIC TABLESPACE MANAGEMENT.................................................................................................11 ......................................................................................................................................11 4.1 Backup method

................................................................................11 4.1.1 Traditional Netcool/Proviso Backup Method ................................................................................11 4.1.2 Alternative Netcool/Proviso Backup Method

Netcool/Proviso Backup Procedures Page 1/24



...................................................................................................11 4.1.3 Restricting Backups by Set Type

..............................................................................................................................12 4.2 Data Channel Impact

.........................................................................................................................................12 4.3 Support Tools .......................................................................................................12 4.3.1 TBSP_BACKUP_DESC table .......................................................................................................13 4.3.2 TBSP_RELOAD_DESC table ....................................................................................13 4.3.3 LDR_TBSP_RELOAD_DESC_VW view ..................................................................................14 4.3.5 DLDR_TBSP_RELOAD_DESC_VW view ...............................................................................................................14 4.3.6 PVM_BACKUP package

..................................................................................................................................................16 5 RECOVERY

..........................................................................................................16 5.1 Database Recovery using RMAN

................................................................................................................................16 5.2 Recovery Scenarios

......................................................................................................................19 6 SAMPLE BACKUP SCRIPTS

.................................................................................................................................19 6.1 Why use Samples?

..................................................................................................................................19 6.2 Sample Script List

..............................................................................................................................19 6.3 Sample script details .......................................................................................................................................19 6.3.1 logon.info ......................................................................................................20 6.3.2 PV_Tbsp_Alter_ReadOnly.ksh .........................................................................................................20 6.3.3 PV_Hot_Skip_Readonly.ksh 6.3.4 PV_Readonly_Tbsp_Backup.ksh...................................................................................................20 .......................................................................................................20 6.3.5 PV_Reset_Backup_Status.ksh .................................................................................................................20 6.3.6 PV_partition_purge.ksh

................................................................................20 7 ESTIMATING SPACE REQUIREMENTS FOR B&R

............................................................................................................................20 7.1 Disk or Tape Backups

..........................................................................................................................21 7.2 RMAN vs OS Backups

....................................................................................................................................................21 7.3 System

...............................................................................................................................21 7.4 PROVISO Metadata

............................................................................................................................................22 7.5 Metric Data ...................................................................................................................22 7.5.1 In Process Metric Data .............................................................................................................22 7.5.2 READ ONLY Metric Data

............................................................................................................................22 7.6 Sample Space Queries ............................................................................................................22 7.6.1 Determine Tablespace Size ..........................................................................................22 7.6.2 Determine Free Space in a Tablespace .......................................................................23 7.6.3 Determine Space Used by Objects in a Tablespace

...................................................................................................................................................23 8 SUMMARY

...................................................................................................................................................23 8.1 Backup ...............................................................................................................24 8.1.1 Required Steps Summary

................................................................................................................................................24 8.2 Recovery




1 Document Overview

This document explains how to perform backups on the Netcool/Proviso DataChannel and database components.

2 Backing Up the DataBase The Netcool/Proviso 4.0 database uses Oracle features that assist loading performance (for example, SQL*Loader in direct path mode with the unrecoverable option.) The use of these features complicates the backup and recovery strategy. To reduce complexity and enhance reliability, procedures have been developed to assist the user with backup and recovery strategy. These procedures should be considered a set of recommendations, requirements and sample code to assist the user in their environment. Many companies have well defined procedures in place to perform backup of their databases. With this in mind, we have attempted to develop a backup procedure that meets Micromuse requirements with the maximum flexibility for the customer. In this document, the difference between required steps and sample steps are detailed.

2.1 Warnings

Backup and recovery is a complex area of database administration. When coupled with the use of unrecoverable direct load methods, it becomes significantly more complex. Make sure that you understand how the system operates and what impact any change will have on the ability to recover the database.

2.2 Reference Material

Micromuse recommends the following reference manuals from Oracle. Oracle 8i database:

Oracle8i Concepts Oracle8i Administrator's Guide Oracle8i Utilities Oracle8i Backup and Recovery Guide Oracle8i Recovery Manager User's Guide and Reference

Oracle 9i database:

Oracle9iR2 Concepts Oracle9iR2 Administrator's Guide Oracle9iR2 Utilities Oracle9iR2 Backup and Recovery Concepts Oracle9iR2 Recovery Manager User's Guide

Online versions of these manuals can be found at http://docs.oracle.com/


http://docs.oracle.com/



2.3 Concepts

2.3.1 Direct Loading / UNRECOVERABLE

In order to obtain maximum performance, Micromuse uses the Oracle SQL*Loader utility in direct path mode with the UNRECOVERABLE option. This use let’s load performance be as fast as possible but it does have its drawbacks. One drawback is that direct path loading with the UNRECOVERABLE option does not generate redo log information for data being loaded. Objects loaded by this method are marked as invalid when a recovery is performed. In order to recover data that has been loaded, but not yet backed up as a READ ONLY tablespace, one of the following two processes must be used:

1. Restore the tablespace from a previous intermediate hot backup 2. Apply all archive logs 3. Revalidate all the objects that were left in an invalid state by the load 4. Reload all the data associated with the objects that were revalidated

If the tablespace has not yet been backed up at all:

1. Drop the tablespace 2. Recreate the tablespace with the same parameters 3. Reload all the associated data ( note the load will automatically create the necessary partitions )

Data must be reloaded in both processes, which means the Netcool/Proviso load components must be involved with recovery operations. The performance gains provided by using direct path with the unrecoverable option far outweigh the complexities that an infrequent recovery operation causes. Before attempting a recovery operation, make sure you fully understand how the system operates and what the requirements for a complete recovery are.

2.4 Terminology and Naming Conventions

2.4.1 Metric and Meta Data

Meta Data: data used by the system to describe/define the data that is measured. Examples include: Formulas (Metrics) Resources (Subelements) Hosts (Elements) Grouping Thresholds

Metric Data: customer data stored for reporting. Examples include: Numeric Raw Data Baseline Threshold Data Resource Aggregations Group Aggregations




2.4.2 DataChannel Components

DataChannel component names include the following:

LDR: produces group aggregations. Also responsible for merging, remerging and loading hourly raw data and all aggregation data, and performing end of period loads for aggregation data.

DLDR: responsible for sort-merging the 24 individual hourly raw data files and loading the merged file into the database

CME: Complex Metrics Engine is the core DataChannel process. It performs resource level data aggregation for a single Collector.

FTE: responsible for moving files from the Collector machine to the CME machine in standard configuration and from the CME to the LDR machine in remote configuration

BOF:Binary Output File Channel: LDR and DLDR operate at the channel level. The LDR sort-merges files that arrive from each

subchannel and loads them into the database Subchannel: a subset of a channel that basically consists of a collector, a FTE and a CME

2.4.3 Channel Table Set Type

Netcool/Proviso uses the following channel table set types:

NRAW Numeric Raw values BASE Baseline Threshold Values 1DRA Resource aggregated values for one day 1WRA Resource aggregated values for one week 1MRA Resource aggregated values for one month 1DGA Group aggregated values for one day 1WGA Group aggregated values for one week 1MGA Group aggregated values for one month

2.4.4 Metric Data Tablespaces

Format

C{Chnl}_{Type}_{AggSet}_{TSpace} with:

{Chnl} The channel set number (from 1 to 8), 2 digits formatted {Type} The tablespace types (4 characters) {AggSet} Aggregation Set number - 3 digits formatted {TSpace} Tablespace identifier defined with a readable date format - YYYYMMDDHH24 Value

0000000000 is used for working tablespaces. Examples

C03_NRAW_000_0000000000 The working tablespace containing the raw data for channel three




C02_1WRA_000_2000090400 The historic tablespace containing weekly data for resource aggregates for the week starting 09-04-2000 at 00.00, aggregation set zero and channel three.

2.4.5 Metric Data Tablespace Datafiles

Format

{DbId}_C{Chnl}_{Type}_{TSpace}_{file#}.dbf with:

{DbId} The Oracle Database name {Chnl} The Channel Number (from 1 to 8), 2 digits formatted {Type} The Channel Table Set Type (4 characters) {TSpace} Tablespace identifier defined with a readable date format - YYYYMMDDHH24

(0000000000 is defined as a working tablespace) {File#} File number of the tablespace

Examples

TEST_C03_1DGA_0000000000_001.dbf The first datafile of the C03_1DGA_0000000000

tablespace in the TEST database

2.5 READ ONLY Tablespaces

Netcool/Proviso uses the Oracle READ ONLY tablespace. When you alter a tablespace to READ ONLY, further modifications cannot be made to the datafiles associated with that tablespace. (Repeated backups of that tablespace become unnecessary.) After you alter the tablespace to READ ONLY, perform a final backup of the tablespace. You can use that same backup to perform a tablespace recovery while the tablespace remains READ ONLY. The tablespace structure is designed to take advantage of the Oracle READ ONLY tablespace. The product creates historical metric data tablespaces based on channel number, data type, and date. After all data for the time period indicated is loaded in the tablespace, it can be altered to READ ONLY mode and backed up once. You do not have to backup that tablespace again unless data needs to be reloaded for that time period. READ ONLY tablespaces issue: When a tablespace is altered to READ ONLY mode, it waits in a transitional state until all transactions in the database started before the command was issued are completed. All transactions in the entire database are effected, regardless of whether these transactions affect the tablespace you are altering to READ ONLY or not. Transactions started after the command was issued do not affect the transition to READ ONLY. Oracle handles these automatically. If an attempt to modify the tablespace in the transitional state is made, it will fail. A change to any other tablespace is allowed and does not affect the edit to READ ONLY transaction.

2.6 Hot Backups

A hot backup means that you perform a backup while the database is open and active. Hot backups require a database in archivelog mode. The Netcool/Proviso default is the database in noarchivelog mode. In order to support




backup and recovery, the customer must alter the database to archivelog mode. The customer must perform this required step after the database they create the database. Refer to Oracle Note:69739.1 on the Oracle support website (http://metalink.oracle.com) for information on how to enable archivelog mode.

2.7 Netcool/Proviso Tablespace Management

Netcool/Proviso separates the metric and meta data into two separate schemas:

PV_METRIC schema contains all metric data PV_ADMIN schema contains all meta data.

Netcool/Proviso has developed tracking tools to monitor and support the system. See for more information. To support the items above, Netcool/Proviso must be involved in the process. Note that Netcool/Proviso loads data in chronological order. Tablespaces (and the partitions that reside in them) are date stamped and tied to the date stamps of the data that resides in them.

2.8 Oracle Recovery Manager (RMAN)

Micromuse recommends you use the Oracle-supplied RMAN backup utility to perform backups. To use RMAN, you must use a RMAN catalog database unless you perform baseline backups on a regular basis. If you decide not to use RMAN, choose a method that meets the Netcool/Proviso requirements.




2.9 Data Channel Loading Summary

This following diagram illustrates the flow of data through the Data Channel. The flow of data within the Data Channel is significantly more complex. Step 1 The Binary Output File (BOF) from a subchannel is placed in the loader “do” directory. In reality, this could actually be several files from multiple subchannels that would merged together by the LDR.

Database Server

Previous Day Tablespace

Data Channel

Loader Directory

done \

state

do

Step 2 The corresponding tablespace is created and the file is loaded using SQL*Loader.

Database Server


Current Day Tablespace

Data Channel

Loader Directory

done \

state

do




Step 3 After the load is completed, the file is moved to the “state” directory.

Database Server



Data Channel

Loader Directory

done \

state

do

Step 4 The file is retained until the tablespace is placed in READ ONLY mode and backed up. This is done by a asynchronous process. Once the tablespace is in READ ONLY and backed up, the OS file can then be deleted.

Database Server



Data Channel

Loader Directory

done \

state

do

2.10 Data Channel Reloads

The DataChannel BOF files must remain available for reload from the time the data is first loaded into the tablespace, and until the tablespace has been placed in READ ONLY mode and backed up. These files are used for restoration purposes in the event of a database failure.




3 Non-Metric Tablespace Management

3.1 Backup Method

The tablespaces that do not contain metric data (for example, Netcool/Proviso Metadata, Rollback, System ) should be backed up using a traditional hot backup strategy. These tablespaces do not have UNRECOVERABLE operations performed on them and should be backed up via this traditional Oracle backup method.

4 Metric Tablespace Management

4.1 Backup method

Metric data tablespaces require different backup strategies that are discussed in the sections below. Each method uses traditional hot backups for metric data tablespaces have not yet been placed in READ ONLY mode, but differ in how they approach the backup of READ ONLY tablespaces. To minimize effort during recovery, backup all metric data tablespaces that have been recently placed in READ ONLY mode first, followed by a traditional hot backup of any metric data tablespaces that are in the process of being loaded.

4.1.1 Traditional Netcool/Proviso Backup Method

The traditional backup method backs up READ ONLY tablespaces once to a more permanent storage area (i.e. a separate disk location or tape pool), where they are retained until the corresponding data is purged. Backing up the READ ONLY tablespaces once minimizes your regular backup window, as there is less data to process on any given backup run. However, this method requires more management overhead, as the backups must be retained and managed using an RMAN catalog database until the data is purged.

4.1.2 Alternative Netcool/Proviso Backup Method

The alternative backup method backs up READ ONLY tablespaces following a periodic reset of READ ONLY tablespaces. Using this method minimizes your regular backup window, as there is less data to process until the next reset, and simplifies management requirements as no RMAN catalog database is required. However, this method requires sufficient backup resources to perform a regular full backup of all your READ ONLY tablespaces.

4.1.3 Restricting Backups by Set Type

No matter which backup method you select, there are some additional options that may help you. There are sample backup scripts that use set types to allow you to reset and backup subsets of the READ ONLY data. When calling the sample scripts, you can specify the following types as options from the command line:

individual set type (NRAW, BASE, 1DRA, 1DGA, 1WRA, 1WGA, 1MRA or 1MGA ) a combination of the 1DRA, 1DGA, 1WRA, 1WGA, 1MRA and 1MGA set types (ALLAGG) a combination of all the set types (ALL)

The ability to restrict to a particular set type or group of set types is useful in several situations:




Scenario 1 You have different data retention requirements for certain set types, and wish to backup these set types to separate (i.e. different tape pools).

Example 1 - You perform a backup using the sample scripts and the default value of ALL (a combination of all the set types). This approach requires you to set the tape retention to the set type that has the longest retention requirement. As you are backing up shorter term (NRAW/BASE) and longer term (aggregated) data to the same tape pool, a large number of tapes must be retained whose data will not be used during a recovery operation.

Example 2 - You perform three backups using NRAW, BASE and ALLAGG options. The NRAW and BASE option allows you to allocate the backups to the same tape pool as they have the same data retention requirements. The ALLAGG option allows you to allocate the backup to a separate tape pool. This approach returns the tapes containing the NRAW and BASE data to their tape pool at the end of their retention window (i.e. 90 days), and the tapes containing the ALLAGG data to their tape pool at the end of their retention window (i.e. 3 years).

Scenario 2 You used the alternative backup method, but do not have sufficient backup resources to complete the backup in the time available. You can reset only a portion of the READ ONLY tablespaces and back up those tablespaces instead.

Example 1 - You have a 10 TB database with roughly 5 TB of raw/base data and 5 TB of aggregated data. Your backup resources allow you to backup 5 TB of data during your backup window, and you want to perform a monthly backup of your READ ONLY tablespaces. In this case, you can reset and backup only the NRAW and BASE data on the 1st rd weekend of the month, then perform a reset and backup of the aggregation data on the 3 weekend of the month. This approach allows you to perform full monthly backups of all READ ONLY tablespaces without requiring additional backup resources.

4.2 Data Channel Impact

By default, the Data Channel component of Netcool/Proviso deletes BOF files for space management. To enable backups, you must edit the Data Channel configuration so the channel components call the CHECK_BACKUP_OF_EOP procedure in PVM_BACKUP before deleting the BOF files necessary for reload operations. The Data Channel configuration parameter SHOULD_CHECK_DB_TO_DELETE_EOP needs to be set to TRUE. See the Data Channel installation instructions for more information on how to set this parameter.

4.3 Support Tools

The following components support the product’s backup and recovery needs:

TBSP_BACKUP_DESC table TBSP_RELOAD_DESC table LDR_TBSP_RELOAD_DESC_VW view DLDR_TBSP_RELOAD_DESC_VW view PVM_BACKUP package

4.3.1 TBSP_BACKUP_DESC table

Netcool/Proviso uses the TBSP_BACKUP_DESC table to track every metric data tablespace created for the following information:




currently in READ ONLY mode backup up occurred after being placed in READ ONLY mode last time that an operation was performed on the tablespace

Name Null? Type ----------------------------------------- -------- ------------ STR_NAME NOT NULL VARCHAR2(32) INT_RO_STATUS NOT NULL NUMBER(1) INT_BU_STATUS NOT NULL NUMBER(1) DTE_DATE NOT NULL NUMBER(10)

When the appropriate Data Channel component (LDR for aggregated data and DLDR for raw data) attempts to load a file after the remerge window is complete, it first calls the ADD_RESIZE_TABLESPACE procedure in the PVM_DATALOAD package.

If the tablespace does not exist, a row is added to the TBSP_BACKUP_DESC table with the current date and the ReadOnly (RO) and BackUp (BU) flags are set to zero.

If the tablespace exists, the record in TBSP_BACKUP_DESC is modified. The date is set to current date and the RO/BU flags are set to zero.

This table is used by subsequent processes to perform actions such as look for all tablespaces that are ready to be placed into READ ONLY mode and look for all tablespaces that have been placed in READ ONLY mode but that have not been backed up yet. This table is also used to prevent BOF files needed for recovery from being deleted by the flow control process in Data Channel.

4.3.2 TBSP_RELOAD_DESC table

Netcool/Proviso uses the TBSP_RELOAD_DESC table to coordinate activity between database recovery and data reloading operations.

Name Null? Type ----------------------------------------- -------- ---------- IDX_CHANNEL NOT NULL NUMBER(2) IDX_TABLE_SET_TYPE NOT NULL NUMBER(2) IDX_AGGSET NOT NULL NUMBER(3) DTE_TIMESTAMP NOT NULL NUMBER(10)

4.3.3 LDR_TBSP_RELOAD_DESC_VW view

The LDR_TBSP_RELOAD_DESC_VW view is based upon the TBSP_RELOAD_DESC table and is used by Netcool/Proviso to coordinate activity between database recovery and data reloading operations performed by the data channel LDR process. This view is scanned during startup and periodically while running. If the LDR finds rows in this view, it moves the files from the done directory to the do directory, deletes the corresponding row from TBSP_RELOAD_DESC (via an API,) and reloads the data into the database.

Name Null? Type ----------------------------------------- -------- -------------- CHNL_NUM NOT NULL NUMBER(2)




TABLE_TYPE NOT NULL CHAR(4) AGGSET_NUM NOT NULL NUMBER(3) DTE_TIMESTAMP VARCHAR2(4000)

4.3.4

4.3.5 DLDR_TBSP_RELOAD_DESC_VW view

The DLDR_TBSP_RELOAD_DESC_VW view is based upon the TBSP_RELOAD_DESC table and is used by Netcool/Proviso to coordinate activity between database recovery and data reloading operations performed by the data channel DLDR process. This view is scanned during startup and periodically while running. If the DLDR finds rows in this view, it moves the files from the done directory to the do directory, deletes the corresponding row from TBSP_RELOAD_DESC (via an API,) and reloads the data into the database.

Name Null? Type ----------------------------------------- -------- -------------- CHNL_NUM NOT NULL NUMBER(2) TABLE_TYPE NOT NULL CHAR(4) AGGSET_NUM NOT NULL NUMBER(3) DTE_TIMESTAMP VARCHAR2(4000)

4.3.6 PVM_BACKUP package

The PVM_BACKUP package is used in many different areas of backup and recovery. The following information describes the components of the API.

Procedure ALTER_TBSP_TO_READONLY

You can call The Alter_Tbsp_To_Readonly procedure through SQL*Plus or by using the dbMgr utility. This procedure examines the TBSP_BACKUP_DESC for all tablespaces for the following information:

tablespace is “full”. All required partitions were placed into the tablespace and have not been touched for the last X hours as defined by DATABASE.BACKUP.READONLY_DELAY registry parameter (default of 2 hours)

registry parameter DATABASE.BACKUP.READONLY_TIMEOUT has expired ( default is 36 hours ) without any changes to the tablespace.

After the tablespace is set to READ ONLY, the corresponding row in TBSP_BACKUP_DESC has its timestamp updated and the RO flag is set to one.

Procedure BACKUP_COMPLETE_FOR_RO_TBSP

This procedure accepts a tablespace name and updates the TBSP_BACKUP_DESC table by setting the BU flag to one. This API that lets you make a call to Netcool/Proviso that notifies Micromuse that the backup of a read only tablespace has been completed.

Function CHECK_BACKUP_OF_EOP

The Data Channel flow control process calls this function to determine if it can delete a BOF file. If the tablespace where the BOF file is destined to reside has both the RO and BU flags set to one, this function returns a flag to Data Channel indicating an OK to delete the BOF file. If the RO and BU flags are not set to one the function returns a flag that tells Data Channel not to delete the file.

Procedure CLEANUP_FILE_RELOAD_TABLE




LDR/DLDR calls this procedure as it scans the TBSP_RELOAD_DESC_VW and moves files for reloading. The API lets LDR/DLDR notify the database that the file has been moved and will be reloaded.

Procedure REVALIDATE_OBJECTS_IN_TBSP

Use this procedure with care (and only when necessary) during a tablespace recovery operation. It accepts a tablespace name as an argument and revalidates the invalid partitions in the specified tablespace. This procedure is important because of an Oracle problem with invalid objects in a partitioned IOT. When you perform an UNRECOVERABLE direct load into a partitioned IOT and need to perform a subsequent recovery on that tablespace, all partitions that were loaded but not yet backed up are left in an invalid state. If a table partition in a partitioned IOT is in an invalid state, you cannot truncate the table (as you can in a regular partitioned table.) You can exchange the partition with a stand-alone table, truncate the stand-alone table, and then exchange it back into the partitioned IOT. The result is the table partition is now empty and valid, and can be reloaded.

The REVALIDATE_OBJECTS_IN_TSP procedure simplifies this operation. The procedure identifies the type of table stored in the tablespace you want to revalidate and creates a dummy non-partitioned table of the same definition. For every invalid table partition in the tablespace, it exchanges the dummy table with the partition, truncates the table, and exchanges the non-partitioned table back into the partitioned IOT. This procedure also populates the TBSP_RELOAD_DESC table with one row for every table partition in the tablespace. This procedure has a second parameter that allows a forced revalidation to be performed. The default value of the force parameter is FALSE, which means this procedure only revalidates and reloads those partitions currently in an invalid state. If you pass a value of TRUE to the force parameter, this procedure revalidates every partition in the tablespace regardless of the partition current state. In most cases, accept the default behavior and only revalidate the partitions that are currently invalid.

Procedure DROP_TBSP_AND_RELOAD

Use this procedure with care (when absolutely needed) during a tablespace recovery operation. It accepts a tablespace name as an argument and drops the specified tablespace. Use this procedure only when you need to reload a tablespace that was created since your last backup. This procedure identifies the type of table stored in the tablespace you want to reload and populates the TBSP_RELOAD_DESC table with one row for every table partition currently in the tablespace. After this process completes, the tablespace is dropped.

Procedure FORCE_TBSP_DROP_WITHOUT_RELOAD

Use this procedure with care (when absolutely needed) during a tablespace recovery operation. It is similar to the DROP_TBSP_AND_RELOAD procedure, but does not attempt to perform a reload of the tablespace once it drops the table partitions. Typically, this procedure is only used in a test or development environment where backups are not implemented, and data loss is tolerable. This procedure identifies the type of table stored in the tablespace, then drops every table partition currently in the tablespace. After this procedure completes, the tablespace is dropped.

Procedure UPDATE_BACKUP_STATUS

Use this procedure (a low level API) to update the RO and BU flags in the TBSP_BACKUP_DESC table.

Procedure PURGE_TBSP_BACKUP_DATA

Use this procedure (a low level API) to remove tablespace backup data from the TBSP_BACKUP_DESC table when tablespaces are purged.

Procedure RESET_RO_TBSP_BACKUP_STATUS




This procedure resets the INT_BU_STATUS of all tablespaces back to zero. This, in turn, forces all read-only tablespaces to be backed up again. Use the procedure when you want to perform a new baseline backup of all your READ ONLY tablespaces, or you can specify a set type option to this function that limits the reset to a particular set type.

Function DETERMINE_METRIC_PURGE_DATE

Use this function (a low level API) to determine the purge date for a given channel, set type, and retention period.

Procedure SYNCHRONIZE_TBSP_BACKUP_DESC

This procedure ensures that the TBSP_BACKUP_DESC table stays synchronized with the appropriate Oracle system dictionary view (DBA_TABLESPACES.) When run, the procedure performs the following modifications:

Deletes a row that exists in TBSP_BACKUP_DESC but not in the data dictionary

Adds a row that exists in the data dictionary but not in TBSP_BACKUP_DESC

Updates TBSP_BACKUP_DESC so the READ ONLY status of the tablespace matches the data dictionary status

Usually this procedure is not necessary but it is useful in some circumstances (for example, a tablespace is manually dropped without using the Netcool/Proviso tools.)

Function VERSION

This function returns the current version for the PVM_BACKUP package. The parameter passed indicates whether to return the version for the package spec (0) or the package body (1).

5 Recovery

5.1 Database Recovery using RMAN

This document does not contain the specifics of how to recover an Oracle database using the RMAN utility. Consult the Oracle documentation for a complete description of RMAN and how to use it during a database recovery

5.2 Recovery Scenarios

The following information contains basic grouping of recovery scenarios:

Recovery of a meta data or Oracle system related tablespace. Examples include:

o SYSTEM o RBS o PV_CFGD o PV_CFGI o PV_CFHD o PV_CFHI o PV_PROH o PV_THRH o PV_REPT o PV_REPM




o PV_LOIS o PV_VOID

To recover these tablespaces, restore the most recent copy of the tablespace using RMAN, recover the tablespace by applying all archive logs generated since your backup was taken. This scenario shows a recovery after you attempt to startup your database and it reports an error similar to the following:

ORA-01157: cannot identify/lock data file 5 - see DBWR trace file ORA-01110: data file 5: '/export/home2/PV/PV_PV_CFHD_0001.dbf'

The following information appears in the alert log:

Errors in file /export/home/oracle/admin/PV/bdump/pv_dbw0_3051.trc: ORA-01157: cannot identify/lock data file 5 - see DBWR trace file ORA-01110: data file 5: '/export/home2/PV/PV_PV_CFHD_0001.dbf' ORA-27037: unable to obtain file status SVR4 Error: 2: No such file or directory Additional information: 3 ORA-1157 signalled during: ALTER DATABASE OPEN...

Log into RMAN, restore and recover the tablespace and then open the database: RMAN> run 2> { 3> allocate channel ch1 type disk; 4> restore tablespace PV_CFHD; 5> recover tablespace PV_CFHD; 6> } RMAN-03022: compiling command: allocate RMAN-03023: executing command: allocate RMAN-08030: allocated channel: ch1 RMAN-08500: channel ch1: sid=9 devtype=DISK RMAN-03022: compiling command: restore RMAN-03022: compiling command: IRESTORE RMAN-03023: executing command: IRESTORE RMAN-08016: channel ch1: starting datafile backupset restore RMAN-08502: set_count=139 set_stamp=446155924 creation_time=18-NOV-01 RMAN-08089: channel ch1: specifying datafile(s) to restore from backup set RMAN-08523: restoring datafile 00005 to /export/home2/PV/PV_PV_CFHD_0001.dbf RMAN-08023: channel ch1: restored backup piece 1 RMAN-08511: piece handle=/export/home5/hot_backup/df_PV_1_139 tag=null RMAN-08024: channel ch1: restore complete RMAN-03022: compiling command: recover RMAN-03023: executing command: recover(3) RMAN-08054: starting media recovery RMAN-03022: compiling command: recover(4) RMAN-06050: archivelog thread 1 sequence 11 is already on disk as file /export/home4/PV/arch/PV_1_11.arch RMAN-06050: archivelog thread 1 sequence 17 is already on disk as file /export/home4/PV/arch/PV_1_17.arch RMAN-03023: executing command: recover(4) RMAN-08515: archivelog filename=/export/home4/PV/arch/PV_1_11.arch sequence=11 RMAN-08515: archivelog filename=/export/home4/PV/arch/PV_1_15.arch sequence=15 RMAN-08055: media recovery complete




RMAN-08031: released channel: ch1 RMAN> run 2> { 3> sql 'alter database open'; 4> } RMAN-03022: compiling command: sql RMAN-06162: sql statement: alter database open RMAN-03023: executing command: sql RMAN>

Recovery of a tablespace that has been placed into READ ONLY mode and backed up While similar to the first scenario, it differs in that you only have to restore the tablespace. No recovery is needed because it has been READ ONLY mode ever since it was backed up.

RMAN> run 2> { 3> allocate channel ch1 type disk; 4> restore tablespace C01_NRAW_000_2001111500; 5> }

RMAN-03022: compiling command: allocate RMAN-03023: executing command: allocate RMAN-08030: allocated channel: ch1 RMAN-08500: channel ch1: sid=9 devtype=DISK RMAN-03022: compiling command: restore RMAN-03022: compiling command: IRESTORE RMAN-03023: executing command: IRESTORE RMAN-08016: channel ch1: starting datafile backupset restore RMAN-08502: set_count=121 set_stamp=446063881 creation_time=17-NOV-01 RMAN-08089: channel ch1: specifying datafile(s) to restore from backup set RMAN-08523: restoring datafile 00104 to /export/home3/PV/PV_C01_NRAW_000_2001111500_001.dbf RMAN-08023: channel ch1: restored backup piece 1 RMAN-08511: piece handle=/export/home6/readonly_backup/df_PV_1_121 tag=null RMAN-08024: channel ch1: restore complete RMAN-08031: released channel: ch1 RMAN> run 2> { 3> sql 'alter database open'; 4> } RMAN-03022: compiling command: sql RMAN-06162: sql statement: alter database open RMAN-03023: executing command: sql RMAN>

Recovery of a tablespace that has data loaded into it but has not been backed up to permanent storage. How you handle this scenario depends on whether you have intermediate backups: If you have an intermediate backup, restore the tablespace and run the REVALIDATE_OBJECTS_IN_TBSP procedure in PVM_BACKUP. (Having the BOF files is




important.) This procedure populates the TBSP_RELOAD_DESC table. LDR/DLDR then scans the corresponding views and reloads the associated files into the table partitions. If you do not have an intermediate backup and you have not yet created any tablespaces newer then the one you want to restore, run the DROP_TBSP_AND_RELOAD procedure in PVM_BACKUP. This procedure drops the tablespace and populates the TBSP_RELOAD_DESC table. LDR/DLDR then scans the corresponding views, creates the tablespace, and reloads the associated files into the table partitions.

6 Sample Backup Scripts

6.1 Why use Samples?

Every location has a different backup philosophy. Some installations are more mature than others. Some need guidance and some require us to fit into their structure. The backup method was developed based on the knowledge that no two locations are alike. Several sample scripts were developed that can be modified for a specific locationt. Or scripts developed at the lcoation can be used as long as Netcool/Proviso requirements are met.

6.2 Sample Script List

The samples developed to support Netcool/Proviso backup are located in the extraTools directory of the shipped product. For example, extraTools/conf, extraTools/backup and extraTools/purgepart. These scripts continue to evolve and newer versions will become available as time goes on. Search the sample scripts for ‘# Customize for given environment’ to make changes for a given location. The list includes the following:

logon.info

PV_Tbsp_Alter_ReadOnly.ksh

PV_Hot_Skip_Readonly.ksh

PV_Readonly_Tbsp_Backup.ksh

PV_Reset_Backup_Status.ksh

PV_partition_purge.ksh

The PV_partition_purge.ksh script needs to be run as the Netcool/Proviso DataMart owner (i.e. pvuser) but the rest of the scripts should be run as the oracle software owner (i.e. oracle).

6.3 Sample script details

6.3.1 logon.info

This file provides a common location of environment information used in the following scripts. It stores information such as usernames and passwords instead if storing them in each script. The following scripts extract the necessary information from the logon.info file. Therefore, a copy of this file must exist and be accessible by the user who is running these scripts.




6.3.2 PV_Tbsp_Alter_ReadOnly.ksh

The ALTER_TBSP_TO_READONLY procedure is called directly through SQL*Plus or through the dbMgr utility. This sample script is a wrapper around the SQL*Plus call of this stored procedure. The script detects whether the previous iteration of the script is still running. As mentioned in an earlier section, an edit to READ ONLY command can hang if there are open uncommitted transactions on the database. (For example, someone updates a row in SQL*Plus and then goes home for the weekend. This sample script exits if the previous run is still active. The script can be customized for specific locations. Examples of customizations include the following:

Sending an email if the previous iteration is still running

Sending a page if the previous iteration is still running

Executing another script if the previous iteration is still running

Changing the number of days worth of logs to keep

If this script, or one like it, place a call to the associated dbMgr command in the crontab. The command should normally be called once or twice per day. For further documentation about the dbMgr utility, see QLAB1026D.

6.3.3 PV_Hot_Skip_Readonly.ksh

The PV_Tbsp_Alter_ReadOnly.ksh script uses Oracle’s RMAN utility to perform a hot backup of all tablespaces that are not in READ ONLY mode.

6.3.4 PV_Readonly_Tbsp_Backup.ksh

The PV_Readonly_Tbsp_Backup.ksh script queries the TBSP_BACKUP_DESC table and dynamically creates the RMAN commands that will perform a tablespace level backup of every tablespace that is in READ ONLY mode but has not been backed up. It also formats the commands that call the BACKUP_COMPLETE_FOR_RO_TBSP procedure. This script then calls the generated script to actually perform the backup. By default, this script backs up all eligible tablespaces, but you can restrict the backup to a particular set type by using the “-st” command line option.

6.3.5 PV_Reset_Backup_Status.ksh

The PV_Reset_Backup_Status.ksh script resets the backup state of the tablespaces stored in the TBSP_BACKUP_DESC table. This script is used in conjunction with the alternative backup method. The script has a “-st” command line option that allows you to specify which set type to reset.

6.3.6 PV_partition_purge.ksh

The PV_partition_purge.ksh script automates the managment of metric data retention in the database. You can call a dbMgr procedure purge data for a given channel and set type, which results in the purging of metric data tablespaces.

7 Estimating Space Requirements for B&R

7.1 Disk or Tape Backups The first decision you will face is whether to backup to disk or to tape. Each approach has advantages and disadvantages:




disk backups allow for a faster recovery but require the purchase of additional disk space direct to tape backups often require a Media Manager to interface with your tape drives, which adds additional

requirements and complexity to the process unless you are already familiar with this area if you backup to disk, often the disks are backed up to tape

Choose a solution that suits your needs. Many locations select a combination of disk and tape backups where a disk backup is performed and a subsequent job backs up those disks to tape. This approach allows you to maintain the most recent backup on disk for quick recoveries while minimizing the space needed for that disk backup.

7.2 RMAN vs OS Backups One advantage that RMAN has over traditional OS backups is that it reduces backup space requirements by “skipping” the portions of the tablespace that are unused, which is sometimes referred to as a compressed backup. RMAN uses the high water mark of the file when determining where it can stop the backup. To illustrate with an example, take a tablespace that is 100M is size but only the first 10M has ever been used. In this case, an OS backup utility will backup the entire 100M file, while RMAN will backup the first 10M and places a marker in the backup file indicating that the file is 100M in size, but that the space between 10 and 100M is empty. The net result is a significantly smaller backup when using RMAN instead of the OS utility. However, when the tablespace is completely full, the RMAN and OS backups will produce backup files of similar size.

7.3 System The components that are typically backed up include:

SYSTEM and RBS tablespaces control files archived redo logs init files

The overhead of backing up these objects is relatively constant once your system reaches a steady state volume. It is approximated by examining their current size and the available free space. When you first start, estimate space requirements equal to the current size of all objects, plus an estimate of the archived redo log volume. Archived redo log space requirements are directly proportional to the rate of change in your particular environment. Note that metric data is loaded directly and will not significantly contribute to the redo volume, while metadata actions contribute to redo volume. The extent to which metadata actions contribute depends upon several factors, such at the frequency of inventories, size of your network, and the frequency of change in your network. Assuming a nightly backup, you need this space for each backup you wish to retain.

7.4 PROVISO Metadata The space required to backup the metadata is relative to the size of the following metadata tablespaces:

PV_CFGD PV_CFGI PV_CFHD PV_CFHI PV_PROH PV_THRH PV_REPT PV_REPM PV_LOIS

Assuming that you are using RMAN, you can estimate the space required for a full backup of these tablespaces using the Oracle data dictionary. Sum up the size of the segments in these tablespaces, or subtract the available free




space in these tablespaces from their current size. Refer below for examples of these queries. If you are using OS backups, you will require space equivalent to the current size of these tablespaces. Metadata increases over time, and so does the space required for backups. Similar to archived redo volume, the space requirement depends on the size of your network and the frequency of change in you environment. Assuming a nightly backup, you need this space for each nightly backup you wish to retain.

7.5 Metric Data

7.5.1 In Process Metric Data The space required to backup the working metric data is relative to the total size of all working tablespaces, plus the most recently loaded non-working metric tablespaces. Working metric data tablespaces end with 0000000000. The size of these tablespaces is dependent on the size of you network, the frequency of collection, the number of collected metrics, how your data and data remerge windows are grouped. An estimate of this volume can be obtained from Micromuse when you have an accurate estimate of your collection volumes. Assuming a nightly backup, you need this space for each backup you wish to retain.

7.5.2 READ ONLY Metric Data You can save a significant amount of space by selecting a particular backup method. A traditional backup method backs up the entire database on a regular basis, and retains a week or two of these backups on tape. As READ ONLY tablespaces are used, these tablespaces only require a single backup once they are placed into READ ONLY mode (assuming you are using the traditional Netcool/Proviso backup method). Therefore, the total space required to backup the READ ONLY metric data is equal to the total size of these tablespaces. An estimate of this volume can be obtained from Micromuse once you have an accurate estimate of your collection volumes. The amount of space required during each nightly backup is relatively small, as only the tablespaces that were most recently loaded need to be backed up. If you use the alternative Netcool/Proviso backup method, the space requirements for READ ONLY metric data will be slightly higher then the traditional Netcool/Proviso method. The total requirements depend on how frequently you perform a reset and backup.

7.6 Sample Space Queries

7.6.1 Determine Tablespace Size The Dba_Data_Files view provides this information:

Select Sum(Bytes)/1048576 From Dba_Data_Files Where Tablespace_Name = '<TABLESPACE_NAME>';

For example:

Select Sum(Bytes)/1048576 From Dba_Data_Files Where Tablespace_Name = 'PV_CFGD';

7.6.2 Determine Free Space in a Tablespace The Dba_Free_Space view provides this information:




Select Sum(Bytes)/1048576 From Dba_Free_Space Where Tablespace_Name = '<TABLESPACE_NAME>';

For example:

Select Sum(Bytes)/1048576 From Dba_Free_Space Where Tablespace_Name = 'PV_CFGD';

7.6.3 Determine Space Used by Objects in a Tablespace The Dba_Segments view provides this information:

Select Sum(Bytes)/1048576 From Dba_Segments Where Tablespace_Name = '<TABLESPACE_NAME>';

For example:

Select Sum(Bytes)/1048576 From Dba_Segments

Where Tablespace_Name = 'PV_CFGD'

8 Summary

8.1 Backup

Ultimately, the customer is responsible for backing up their data and for ensuring they can recover it in case of failure. Though Micromuse has provided several tools that simplify and assist in the task, an experienced database administrator should review this document and develop the final backup strategy for their specific installation. Backups must be monitored and tested to ensure proper recovery of the database. Even if users decide not to perform backups, they must still meet the requirements defined above.




8.1.1 Required Steps Summary

The customer must call dbMgr alterTablespaceReadOnly at least once per day . Micromuse suggests they use a script similar to PV_Tbsp_Alter_ReadOnly.ksh

The customer does have flexibility about how to perform their backups and what tools they use, but they must notify Netcool/Proviso by using the BACKUP_COMPLETE_FOR_RO_TBSP procedure in PVM_BACKUP. If they fail to do this, EOP BOF files will not be deleted and the LDR/DLDR directories will fill up.

If you use the traditional Netcool/Proviso backup method, then you must use a RMAN catalog database When enabling backups, remember to modify the Data Channel configuration so that it calls the

CHECK_BACKUP_OF_EOP procedure during normal operations.

8.2 Recovery

The recovery of an Oracle database that is in an inconsistent state is not an everyday task. But, when the need arises, you must be confident in your backup. This is also not a task that just anyone can perform. Oracle recovery scenarios are quite extensive. An experienced Oracle database administrator should be able to analyze the situation and perform the appropriate steps to recover your database.


TECHNICAL NOTE Sender: Recipient: Attached Files: Number ......2.4.4 Metric Data Tablespaces Format...

Documents

Transcript of TECHNICAL NOTE Sender: Recipient: Attached Files: Number ......2.4.4 Metric Data Tablespaces Format...