12837GC10 - Oracle9i：Real Application Clusters

Oracle9i: Real Application Clusters

Student Guide

12837GC10Prod 1.0January 2002D34316

Copyright © Oracle Corporation, 2002. All rights reserved.

This documentation contains proprietary information of Oracle Corporation. It is provided under a license agreement containing restrictions on use and disclosure and is also protected by copyright law. Reverse engineering of the software is prohibited. If this documentation is delivered to a U.S. Government Agency of the Department of Defense, then it is delivered with Restricted Rights and the following legend is applicable:

Restricted Rights Legend

Use, duplication or disclosure by the Government is subject to restrictions for commercial computer software and shall be deemed to be Restricted Rights software under Federal law, as set forth in subparagraph (c)(1)(ii) of DFARS 252.227-7013, Rights in Technical Data and Computer Software (October 1988).

This material or any portion of it may not be copied in any form or by any means without the express prior written permission of Oracle Corporation. Any other copying is a violation of copyright law and may result in civil and/or criminal penalties.

If this documentation is delivered to a U.S. Government Agency not within the Department of Defense, then it is delivered with “Restricted Rights,” as defined in FAR 52.227-14, Rights in Data-General, including Alternate III (June 1987).

The information in this document is subject to change without notice. If you find any problems in the documentation, please report them in writing to Education Products, Oracle Corporation, 500 Oracle Parkway, Box SB-6, Redwood Shores, CA 94065. Oracle Corporation does not warrant that this document is error-free.

Oracle and <list copyrights here> are trademarks or registered trademarks of Oracle Corporation.

All other products or company names are used for identification purposes only, and may be trademarks of their respective owners.

AuthorDavid Austin

Technical Contributors and Reviewers

Mark BauerTammy BednarJack CaiMichael CebullaSashikanth ChandrasekaranJonathan CreightonNorbert DebesJoel GoodmanArturo GutierrezBill KehoeTamas KerepesRoland KnappVijay LunawatBarb LundhildPeter SharmanTak Wang

PublisherChristine Markusic

Preface

IntroductionOracle9i: Real Application Clusters I-1Overview I-2What Is a Cluster? I-3What Is Oracle9i Real Application Clusters? I-4Optional Schedule I-5Student Preface I-6

1 ArchitectureObjectives 1-2Cluster Hardware Components 1-3Nodes 1-4Interconnect 1-6Shared Disk Subsystem 1-7Cluster Software 1-9Cluster Manager 1-10Interprocess Communication Software 1-11Real Application Clusters Components 1-12Disk Access 1-13Global Services Daemon 1-15Background Processes 1-16Global Enqueue Service Monitor 1-17Global Enqueue Service Daemon 1-18Global Cache Service Processes 1-19LCK Process 1-20Diagnosability Daemon 1-21Practice Overview: Identifying Components of Real Application Clusters 1-22Summary 1-23

2 Installation and ConversionObjectives 2-2Preliminary Cluster Preparations 2-3Cluster Files or Partitions for Default Database 2-5Global Services Daemon 2-6Information Repository for the Database Server Configuration 2-7Installation Options 2-8OUI Database Configuration Types 2-9Additional Installation Considerations 2-10

Contents

iii

Migrate to Oracle9i Real Application Clusters 2-11Convert a Single Instance to Real Application Clusters 2-12Configure Hardware 2-13Evaluate Tablespace Requirements 2-14Evaluate Log File Requirements 2-15Create Shared File System or Raw Devices 2-16Extract Data from Old Database 2-17Installation Steps 2-18Create the Database 2-19Load Data into New Database 2-21Adjust Parameters 2-22Start the Database 2-23Additional Migration Considerations 2-24Practice Overview: Modifying a Database to Support a Second Instance 2-25Summary 2-26

3 Management and Configuration ToolsObjectives 3-2GSD Management 3-3Server Control Utility 3-4SRVCTL Command Syntax 3-5SRVCTL Cluster Database Configuration Tasks 3-6Add and Delete Databases 3-7Add and Delete Instances 3-8Move and Rename Instances 3-9Store and Remove Environment Data 3-10SRVCTL Cluster Database Tasks 3-11Starting Databases and Instances 3-12Stopping Databases and Instances 3-14Inspect Status of Cluster Database 3-15Inspect Database Configuration Information 3-16Enterprise Manager and Cluster Databases 3-17Displaying Objects in the Navigator Pane 3-18Starting a Cluster Database 3-19Stopping a Cluster Database 3-20Status Details Tab 3-21Viewing Cluster Database Status 3-22Job Management for a Cluster Database or Instance 3-24Registering Cluster Database Events 3-25Parameter Files in Cluster Databases 3-26Create and Manage Server Parameter File 3-27Parameter File Search Order 3-28Practice Overview: Employing Database Configuration and Mgt. Tools 3-29Summary 3-30

iv

4 Scalability and Cache FusionObjectives 4-2Levels of Scalability 4-3Scaleup and Speedup 4-3Speedup and Scaleup for Different Types of Workloads 4-5Load Balancing with Oracle Net Services 4-6Client Load Balancing 4-7Connection Load Balancing 4-8Example of Connection Load Balancing 4-9Service and Instance Names 4-10Adaptive Parallel Query 4-11Cache Fusion 4-12Cache Fusion Model 4-13 Global Cache Service Resource Modes 4-14Global Cache Service Resource Roles 4-15Cache Fusion Block Transfers: Example Overview 4-16Example 1: Read with No Transfer 4-17Example 2: Read to Write Transfer 4-21Example 3: Write to Write Transfer 4-25Example 4: Write to Read Transfer 4-29Block Transfers in Real Application Clusters 4-33Configure Multiblock Locks 4-34Notes on Using Multiblock Locks 4-35Practice Overview: Working Across Multiple Instances 4-36Summary 4-37

5 High Availability ConsiderationsObjectives 5-2High Availability Features 5-3Cache Resource Remastering 5-4Automatic Resource Remastering 5-5LMON and Cluster Reorganization 5-6Example 5-7Instance Transition and Recovery 5-9Global Resource Directory Reconfiguration and Cache Recovery 5-10High Availability Design Considerations 5-11Change Management 5-12Configure Redundant Hardware 5-13Primary and Secondary Instances 5-14Spare Nodes 5-15Parameters for High Availability 5-16Oracle9i Data Guard 5-19Standby Database Real Application Clusters Support 5-20Log Transport Services: Standby Database Set Up 5-21Log Transport Services: Primary Database Setup 5-22

v

Example 5-23Switchover Considerations 5-24Real Application Clusters Guard Architecture 5-26Real Application Clusters Guard Failover 5-27Practice Overview: Tuning Instance Recovery 5-28Summary 5-29

6 Transparent Application Failover Objectives 6-2Listeners 6-3TNS Connect Descriptors 6-4Network Naming Methods 6-5Oracle Call Interface 6-5Host-Based Failover 6-7Real Application Clusters Failover 6-8TAF in Real Application Clusters 6-9TAF Failover Mode Options 6-10Failover Types 6-11Failover Methods 6-12Failover Backup Service 6-13Failover Connection Reattempt Options 6-14TAF Implementation 6-15Connect-Time Failover with Client Load Balancing 6-16Retrying Failover Connections 6-17Preestablishing a Connection for TAF 6-18TAF Verification 6-19Simple Primary/Secondary Configuration 6-20Primary/Secondary Configuration with TAF 6-21Example 6-22Practice Overview:Configuring TAF 6-24Optional Practice Overview: Configuring TAF 6-25Summary 6-26

7 Backup and Recovery Objectives 7-2Protecting Against Media Failure 7-3Log History Records 7-4Initiate Archiving 7-5Archived Log File Configurations 7-6Oracle Recovery Manager 7-7Configuring RMAN 7-8User-Managed Backup Methods 7-9Offline User-Managed Backup 7-10Restoring and Recovering Redo Log Files 7-12Parallel Recovery in Real Application Clusters 7-13Fast-Start Parallel Rollback in Real Application Clusters 7-15Practice Overview: Backing Up and Recovering Data 7-16Summary 7-17

vi

8 Online Transaction Processing ConsiderationsObjectives 8-2Extent Management Options 8-3Locally Managed Tablespaces 8-4Dictionary Managed Tablespaces 8-5Free Block Management 8-6Automatic Segment-Space Management 8-7Enable Automatic Segment-Space Management 8-8Free List Space Management 8-9Free List Groups 8-10Manual Allocation of Free List Groups to Instances 8-11Automatic Allocation of Free List Groups to Instances 8-12Comparison of Free Space Management Methods 8-13Sequential Numbers 8-14Sequence Generators 8-15Sequence Generator Options 8-16Index Leaf Block Contention 8-17Advanced Queuing Considerations 8-19Queue Table Instance Affinity 8-20Reduce Queue Table Cache Service Resource Requests 8-21Transaction Processing in Earlier Releases 8-22Summary 8-23

9 Considerations for Online Analytical Processing, DSS, and Data WarehousingObjectives 9-2Query-Intensive Database Issues 9-3Populating the Data Warehouse 9-4Populating the Warehouse 9-5Data Loading 9-6Denormalization 9-8Materialized Views 9-9Summary Tables 9-10Index Creation 9-12Partitioning Option 9-13Datamarts 9-14Automatic Control of Parallel Processing 9-15Adaptive Multiuser Feature Used in Parallel Execution 9-16Automatic Parallel Execution Optimization 9-17Manual Control of Parallel Processing 9-18Additional Parallel Processing Parameters 9-19Temporary Tablespaces 9-20Summary 9-21

vii

10 Monitoring and Tuning Objectives 10-2Create Real Application Clusters Views 10-3Oracle Performance Manager 10-4Activate Oracle Performance Manager 10-5Statspack 10-6Configure Statspack 10-7Collect Statspack Statistics 10-8Generate Statspack Report 10-9UTLBSTAT and UTLESTAT 10-10Monitor Consistent Block Processing 10-11Global Cache Service Request Latency 10-12Monitor Current Block Processing 10-14Monitor Block Mode Conversions 10-15Analyze Global Enqueue Statistics 10-16Collect Global Enqueue Statistics 10-17Calculate Average Global Enqueue Time 10-18Calculate Average Global Lock Convert Time 10-19Further Analysis with V$LIBRARYCACHE and V$ROWCACHE 10-21Monitor Global Enqueue Service Resource Statistics 10-22Analyze Global Enqueue Service Resource Statistics 10-23Additional Views to Examine Resource Activity 10-24The V$SYSTEM_EVENT View 10-25Real Application Clusters Events in V$SYSTEM_EVENT 10-26General Observations for Tuning Inter-Instance Performance 10-27Tuning Strategy 10-28Summary 10-29

11 Case Study (Optional) Objectives 11-2Case Study 11-3Summary 11-5

Appendix A

Appendix B

viii

Preface

Preface - 2

Preface - 3

Profile

Before You Begin This CourseBefore you begin this course, you should have the following qualifications:

• Thorough knowledge of Oracle database administration.• Working experience with current releases of Oracle database software.

Prerequisites • Oracle9i Database Administration Fundamentals I (D11321GC11)• Oracle9i Database Administration Fundamentals II (D11297GC11) • Oracle9i Database Performance Tuning (D11299GC11)

How This Course Is OrganizedOracle9i: Real Application Clusters is an instructor-led course featuring lecture and hands-on exercises. Online demonstrations and written practice sessions reinforce the concepts and skills introduced.

Preface - 4

Related Publications

Additional Publications• System release bulletins• Installation and user’s guides• read.me files• International Oracle User’s Group (IOUG) articles• Oracle Magazine

Preface - 5

Typographic ConventionsTypographic Conventions in Text

Convention Element ExampleBold italic Glossary term (if there is

a glossary)The algorithm inserts the new key.

Caps andlowercase

Buttons,check boxes,triggers,windows

Click the Executable button.Select the Can抰 Delete Card check box.Assign a When-Validate-Item trigger to theORD block.Open the Master Schedule window.

Courier new,case sensitive(default islowercase)

Code output,directory names,filenames,passwords,pathnames,URLs,user input,usernames

Code output: debug.set (慖? 300);Directory: bin (DOS), $FMHOME (UNIX)Filename: Locate the init.ora file.Password: User tiger as your password.Pathname: Open c:\my_docs\projectsURL: Go to http://www.oracle.comUser input: Enter 300Username: Log on as scott

Initial cap Graphics labels(unless the term is aproper noun)

Customer address (but Oracle Payables)

Italic Emphasized words andphrases,titles of books andcourses,variables

Do not save changes to the database.

For further information, see Oracle7 ServerSQL Language Reference Manual.

Enter [email protected], whereuser_id is the name of the user.

Quotationmarks

Interface elements withlong names that haveonly initial caps;lesson and chapter titlesin cross-references

Select 揑nclude a reusable module component�and click Finish.

This subject is covered in Unit II, Lesson 3,揥 orking with Objects.�

Uppercase SQL column names,commands, functions,schemas, table names

Use the SELECT command to view informationstored in the LAST_NAMEcolumn of the EMP table.

Preface - 6

Typographic Conventions in Code

Typographic Conventions in Navigation PathsThis course uses simplified navigation paths, such as the following example, to direct you through Oracle Applications. (N) Invoice—>Entry—>Invoice Batches Summary (M) Query—>Find (B) Approve

This simplified path translates to the following:1. (N) From the Navigator window, select Invoice—>Entry—>Invoice Batches Summary.2. (M) From the menu, select Query—>Find.3. (B) Click the Approve button.

N = Navigator, M = Menu, B = Button

Convention Element Example

Arrow Menu paths Select File—> Save.

Brackets Key names Press [Enter].

Commas Key sequences Press and release keys one at a time:[Alternate], [F], [D]

Plus signs Key combinations Press and hold these keys simultaneously:[Ctrl]+[Alt]+[Del]

Convention Element Example

Caps andlowercase

Oracle Formstriggers

When-Validate-Item

Lowercase Column names,table names

SELECT last_nameFROM s_emp;

Passwords DROP USER scottIDENTIFIED BY tiger;

PL/SQL objects OG_ACTIVATE_LAYER (OG_GET_LAYER (�prod_pie_layer? )

Lowercase italic Syntax variables CREATE ROLE role

Uppercase SQL commandsand functions

SELECT useridFROM emp;


Oracle9i: Real Application Clusters

Introduction

Oracle9i: Real Application Clusters I-2

I-2 Copyright © Oracle Corporation, 2002. All rights reserved.

Overview

• This course is designed for anyone interested in implementing a Real Application Clusters database– Hardware and software planning– Application and database design– System and database administration

• The coverage is general and contains platform specific information only when necessary to complete an example

• Knowledge and experience with Oracle9iarchitecture is assumed

• Lecture material is supplemented with hands-on practices

OverviewThe material in this course is designed to provide the basic information needed to help plan or manage an Oracle9i database with the Real Application Clusters option. To provide for the needs of a wide audience, the course is generic and does not contain specific hardware or operating system details. These details are available from the platform-specific documentation provided with your version of Oracle9i Real Application Clusters. You can also check with Oracle University for specific training courses related to your platform. The lessons and practices are designed to build on your knowledge of Oracle used in a nonclustered environment. The material does not cover basic architecture and database management; these topics are addressed by the Oracle9i database administration courses offered by Oracle University. If your background does not include work with a current release of Oracle database, you should consider taking this training before attempting this course.The practices provide an opportunity for you to work with the features of the database that are unique to Real Application Clusters. However, limitations of the classroom environment will not permit every feature to be explored in these practices.



What Is a Cluster?

• Interconnectednodes

• Cluster software hides thestructure

• Disks availablefor read andwrite by nodeor nodesrequiring the data theycontain

Node

Disks

Interconnect

What Is a Cluster?A cluster consists of two or more independent, but interconnected, servers. A number of hardware vendors have provided cluster capability over the years to meet a variety of needs. Some clusters were only intended to provide high availability by allowing work to be transferred to a secondary node if the active node fails. Others were designed to provide scalability by allowing user connections or work to be distributed across the nodes.Another common feature of a cluster is that it should appear to an application as if it were a single server. Similarly, management of the servers should be as similar to the management of a single server as possible. Cluster management software helps provide this transparency.In order for the nodes to act as if they were a single server, files must be stored in such a way that they can be found by the specific node that needs them. There are a number of different cluster topologies that address the data access issue, each dependent on the primary goals of the cluster designer.



What Is Oracle9i Real Application Clusters?

• Database withinstances onseparate nodes

• Physical orlogical accessto each database file

• Software controls access todata

Database files

Instance on each node

What Is Oracle9i Real Application Clusters?Real Applications Clusters is an Oracle9i software option that allows you take advantage of clustered hardware by running multiple instances against a database. The database files are stored on disks that are either physically or logically connected to each node so that every active instance can read or write to them. The Real Application Clusters software manages data access so changes are coordinated between the instances and each instance sees a consistent image of the database. The cluster interconnect enables instances to pass coordination information and data images between each other.Oracle9i Real Application Clusters replaces clustered database options available in earlier releases. It offers transparent scalability, high availability with minimal down time following an instance failure, and centralized management of the database and its instances.



Optional Schedule

Topics

Architecture and management

Database and data availability

Design and tuning

Lessons

1 - 4

5 - 8

9 - 12

Day

1

2

3

Optional ScheduleThe lessons in this guide are arranged in the order that you will probably learn them in class and are grouped into the topic areas shown in the table above. The individual lessons are ordered such that they lead from more familiar to less familiar areas. The related practices are designed to let you explore increasingly powerful features of a Real Application Clusters database. In some cases, the goals for the lessons and those for the practices are not completely compatible. Your instructor may, therefore, choose to teach some of material in a different order than found in this manual. However, if your instructor teaches the class in the order the lessons are printed in this guide, the class should run approximately as shown in the above schedule.



Student Preface

• The material in this manual and your instructor are provided for the benefit of you and your fellow students

• Be prepared to accept new ideas, but do not forget the information with which you arrived

• Be prepared to explore beyond the limits of the class– Try to answer your own questions by exploring the

software available during the practice sessions– Read the Real Application Cluster manuals

Student PrefaceThis class provides you an opportunity to learn about a new piece of Oracle software. You should be prepared to take advantage of the resources made available to you such as this student guide, your instructor, and the hardware and software provided for hands-on practice sessions. However, you should remain aware of the needs of your fellow students:

• Do not monopolize the class time by asking questions unrelated to topic being presented.• During practices, avoid interfering with the databases and the work of other students.• Try to be on time and return promptly from breaks.• Be courteous with your use of pagers and cell phones.

In addition to the structured classroom activities, be prepared to put in some additional time and effort to maximize your training experience. For example, if there is time, you could perform additional steps during practice sessions to try to find answers to some of your questions. You can also look at the following books from the documentation set:

• Oracle9i Real Application Clusters Concepts• Oracle9i Real Application Clusters Installation and Configuration• Oracle9i Real Application Clusters Administration• Oracle9i Real Application Clusters Deployment and Performance• Oracle Real Application Clusters Guard Administration and Reference Guide


Architecture

Oracle9i: Real Application Clusters 1-2

1-2 Copyright © Oracle Corporation, 2002. All rights reserved.

Objectives

After completing this lesson, you should be able to do the following:• Describe cluster architecture components• Identify Oracle components exclusive to Real

Application Clusters



Cluster Hardware Components

• Nodes• Interconnect• Shared disk subsystem

Cluster Hardware ComponentsThe three main components of a hardware cluster—nodes, the interconnect, and a logically or physically shared disk subsystem—are described in detail on the following pages.



Nodes

• Each node is a self-contained server

• It may be a single CPU system or a multiple-CPU system– SMP– NUMA

• Two or more nodes are required for a cluster– Nodes do not have to be

identical– Identical nodes are

recommended for fail over and load balancing

Node

NodesEach node in a cluster consists of one or more central processing units (CPUs), memory, and, possibly, one or more local storage devices. Nodes must also be able to connect with each other node and with a subsystem of shared storage devices. The computing nodes that participate in clusters can typically function as independent servers using their local storage devices and standard network connections for access by client applications.Single CPU nodes are known as uniprocessors. Computers with multiple CPUs can be configured with either uniform or nonuniform access to shared memory. In uniform memory access (UMA) configurations, each CPU has equal access to all of the memory. These systems are known as symmetric multi-processor (SMP) systems. In nonuniform memory access (NUMA) systems, each CPU has a pool of local memory it uses by preference, but can also access the memory assigned to other CPUs, if needed. The nodes used in a cluster do not all have to be the same. For example, some could be uniprocessors and others could be NUMA computers. However, unless you have determined that your application will be partitioned across different nodes, you should plan to use identical nodes for a Real Application Clusters database. This simplifies your application design and allows you to use load balancing and failover techniques with a greater degree of confidence that all users will perceive identical levels of performance.



Nodes

• Each node supports one instance of a Real Application Clusters database

• The instance on a node can be configured for a specific type of activity

• All instances can be configured for the same performance characteristics

• Nodes, and instances, can be added and dropped within certified limits

Nodes (continued)You can run a single database in a cluster, using each node for a different instance. You can also run multiple databases in a cluster using each node to support an instance from just one of these databases. Alternatively, you can run instances from two or more databases on one node.If you have an application that has different configuration requirements for distinct sets of users or application routines, you can initialize separate instances with different parameter settings. You can also choose to define each instance with the same values for the tuning parameters to allow users to log into whichever instance is most appropriate at the time they connect to the database. The total number of nodes in a cluster supporting a Real Application Clusters database depends on the combination of hardware and operating systems. Some systems restrict you to a low power of two for the maximum number of nodes; others can scale beyond a hundred nodes. Various vendors and Oracle continue to work together to certify larger and more powerful clusters using the latest hardware, operating systems, and Oracle releases. Consequently, there is no single upper limit on the number of supported Real Application Clusters nodes. Your Oracle Corporation contact can provide the current certification status of your system.



Interconnect

• Links the nodes• Can use a standard

network protocol such as TCP/IP

• Best results are achieved using high speed interconnects– UDP– VIA– Vendor-specific, certified,

proprietary systems

Interconnect

InterconnectThe cluster interconnect provides a physical link between the nodes of the cluster. The interconnect routes messages between nodes of the cluster. These include cluster-specific synchronization messages as well as database messages, for example, between parallel process servers on different nodes. Cache Fusion also uses the interconnect to transfer database block images between instances on different nodes. Best results are achieved using high speed interconnects such as:

• GigaBit Ethernet• Virtual Interface Architecture (VIA) • Vendor-specific, certified, proprietary systems.

Whatever interconnect you use, you should consider adding a backup interconnect, if not already provided by your vendor, to provide cluster and database integrity should the primary interconnect fail. You can use a standard network protocol such as Transmission Control Protocol/Interconnect Protocol (TCP/IP) or User Datagram Protocol (UDP), or any vendor-provided lightweight protocol that offers the required speed across the interconnect.



Shared Disk Subsystem

Shared devices on a shared disk subsystem • Traditionally

– Raw device volumes on UNIX

– Logical drives on Windows

• Proprietary, vendor-certified, cluster file systems– Have always existed– Becoming more

commonShared disk subsystem

Shared Disk SubsystemA Real Application Clusters database must be accessible from instances running on separate nodes in the cluster. Because most standard file systems do not allow processes from multiple servers to open files for read and write access at the same time, you need to use a disk management system that allows access to the database files across the cluster.A number of file system options are available for use with Real Application Clusters. On some systems, you may use a proprietary cluster file system if it has attained Oracle certification by the cluster vendor. Although such proprietary systems have always existed for prior releases of cluster database software, more vendors are exploring the option to certify alternative shared file systems. On systems without clustered file systems, you will typically use raw partitions (on UNIX servers) or unformatted disk partitions (on Windows). On some hardware platforms, all disks are attached to specific nodes in the cluster rather than being shared across the cluster (known as shared-nothing systems). In order to use Oracle’s cluster database option, disks on shared nothing systems are accessed through an additional layer of software that enables them to behave like shared disks.



Shared Disk Subsystem

• Provides simultaneous access by all nodes to disks used by the cluster software and Oracle database

• Typically implemented with raw (unformatted) disk partitions

• Different shared disk architectures have been, or are becoming, certified

• nonuniform disk access systems can also be supported

Shared Disk Subsystem (continued)Previous versions of Oracle clustered database software required the use of raw partitions (known as unformatted disk partitions on Windows) on all but a few platforms in order to share database files. A number of vendors are working with Oracle to certify their own shared file management systems to support Real Application Clusters.Most systems use uniform disk access for Oracle databases. This means that the files reside on disks that have identical physical connections to each node in the cluster. However, a Real Application Clusters database can reside on a nonuniform disk access cluster. On such clusters, the database files exist on disks that are local to the nodes and a layer of software provides directory services and access to the files across the whole cluster. As with proprietary physical file structures, nonuniform disk access software must be certified by Oracle for use with Real Application Clusters databases.



Cluster Software

The OSD clusterware• Provides cluster

services• Provides access for

Real Application Clusters to the system and cluster services

• Can be supplied by the vendor or by Oracle

OSDClusterware

OSDClusterware

Cluster SoftwareReal Application Clusters software uses all the components of single instance Oracle environments plus cluster software that facilitates internode communication. The operating system cluster software contains operating system-dependent (OSD) clusterware components. The OSD clusterware controls the operating system and clusterware services required for Real Application Clusters. For UNIX operating systems, your vendor provides the OSD clusterware. For Windows operating systems, the OSD clusterware is provided by Oracle as part of the preinstallation software provided with the Oracle9i Enterprise Edition.



Cluster Manager

• Maintains a global view of the cluster– Manages cluster

messaging– Controls cluster

membership• The Node Monitor

component monitors node status

OSDClusterware

ClusterManager

OSDClusterware

ClusterManager

Nodemonitor

Nodemonitor

Cluster ManagerA layer of software, known as the Cluster Manager, provides a clusterwide view of the nodes. The Cluster Manager regulates cluster membership and manages messages sent between nodes. The Node Monitor, a component of the Cluster Manager, monitors the status of the nodes (whether they are active members of the cluster).The Cluster Manager is usually provided by the cluster vendor but Oracle provides this software for Windows environments. Vendor-supplied Cluster Managers can perform special cluster-related tasks that are specific to the system. Real Application Clusters interacts with the Cluster Manager for its own cluster information and requirements.



Interprocess Communication Software

• Is the second main subcomponent of the OSD clusterware

• Controls messages across the interconnect

• Transfers messages and data blocks for Real Application Clusters

OSDClusterware

ClusterManager

Nodemonitor

IPC

OSDClusterware

ClusterManager

Nodemonitor

IPC

Interprocess Communication SoftwareThe interprocess communication (IPC) software is the second key subcomponent of the operating system-dependent OSD clusterware. The IPC layer controls messaging functions so the nodes can communicate with each other through the interconnect.Real Application Clusters enlists the IPC to transfer messages and data blocks between instances on different nodes.



Real Application Clusters Components

• A Real Application Clusters database consists of– Database files– One or more instances

• These components are similar to single-instance databases with some additions

Instance

OSDClusterware

Real Application Clusters ComponentsA Real Application Clusters database consists of the following components you should recognize from a single instance database:

• One or more copies of a control file• A set of online redo log files• An optional, but recommended, set of archive log files• A set of data files• An optional server parameter file (SPFILE)• An SGA for the instance consisting of configurable areas, including:

- The shared pool- One or more database buffer caches- A redo log buffer- An optional large pool- An optional Java pool

• A set of background processes such as database writer (DBW0), log writer (LGWR), process monitor (PMON), system monitor (SMON), archiver (ARC0), and so on.

Additions for Real Application Clusters databases are covered in the upcoming pages.



Disk Access

Shared disk access is required for • Control files• Data files• Online redo log files• A quorum (or voting)

disk on some systems• Configuration data• Optionally, a server

parameter file (SPFILE)

Instance

OSDClusterware

Connectionsto othernodes

Disk AccessA key difference between a Real Application Clusters database and a single-instance database is the requirement for shared disk access. In a non-Real Application Clusters database, only one instance needs access to the database files at a time so these files can be located on disks with the fastest access path from the database server, whether these are internal to the server or in an external disk array. In a Real Application Clusters database, multiple instances need read and write access to the database files so they must be stored on the shared disk subsystem that is available to all of the nodes. To support a Real Application Clusters database, you need a shared disk resource for:

• Each copy of the control file• Every data file• Each online redo log member of every redo log group• A configuration file used by certain Oracle9i tools to manage the database and its

instances.On some platforms, a quorum disk (also known as a voting disk on some systems) is used by the node monitor to manage the cluster configuration. On Windows, the voting disk and configuration data share the same disk resource.If you employ a server parameter file (SPFILE), this must also be shared by the instances.



Disk Access

Optional features requiring shared disk access include:• Configuration

information for management tools

• Server parameter file• Separate undo

tablespace for each instance if using automatic undo management


Instance

OSDClusterware

Disk Access (continued)You will need to provide shared disk access to use certain optional features with your Real Application Clusters database.Node, database, and instance information needed by various management tools, such as the Database Configuration Assistant, the SRVCTL utility, and Oracle Enterprise Manager, must be available across the cluster. You will need a shared disk resource to store this configuration information if you want to use these tools. For those of you running Real Application Clusters in a Windows environment, your voting disk and configuration information will share the same disk resource.Managing your Real Application Clusters database will be easier if you create a server parameter file. The server parameter file is stored on a shared disk resource, however, not on the individual servers as is the case with non-Real Application Clusters databases.If your database uses automatic undo management, a recommended configuration, you also need a separate undo tablespace for each instance. The data files for these tablespaces must be created on shared disk resources.



Global Services Daemon

• Coordinates with other utilities and tools to manage Real Application Clusters databases and instances

• One GSD supports all databases on a node

• Requires no configuration

OSDClusterware

Instance

GSD

Global Services DaemonThe Global Services Daemon (GSD) performs manageability tasks, such as instance startup and shutdown, for clients such as the Server Control Utility (SRVCTL), the Database Configuration Assistant (DBCA), and Oracle Enterprise Manager. You must start the GSD on all the nodes in your Real Application Clusters database so that the manageability features and these tools operate properly. However, you only need one GSD service on each node regardless of how many Real Application Clusters database installations you have.Note: These tools are covered in more detail in later lessons.For a simple example, suppose you start an instance using Oracle Enterprise Manager. The Intelligent Agent launches a script that contains SRVCTL commands. The GSD executes the commands required for the startup operation on the designated node. As another example, assume you execute a SRVCTL command to stop all active instances in your Real Application Clusters database. The GSD receives your request from SRVCTL and executes the command locally on each node. Finally, the GSD returns the results to your SRVCTL session.



Background Processes

Additional background processes for a Real Application Clusters database consist of• LMON• LMD• LMSn• LCK• DIAG

InstanceLMS0LMON

LCK

LMD

DIAG

OSDClusterware

Background ProcessesYou may see a few more additional background processes associated with a Real Application Clusters instance than you would with a single instance database. The processes, described in detail in the following pages, will be among the following:

• LMON: The Global Enqueue Service Monitor• LMD: The Global Enqueue Service Daemon, typically appearing with the process name

ora_lmd0_instance on a UNIX system• LMSn: The Global Cache Service Processes, where n can range from zero to nine

(LMS0 to LMS9)• LCK process: Typically appearing with the process name ora_lck0_instance on a UNIX

system• DIAG: The Diagnosability Daemon



Global Enqueue Service Monitor

• Monitors global enqueues and resources across the cluster

• Performs recovery operations for cluster enqueues

InstanceLMS0LMON

LCK

LMD

DIAG

OSDClusterware

Global Enqueue Service MonitorAn enqueue is a memory structure that serializes access to database resources. Global resources include locks and enqueues that need to be seen by all instances in a Real Application Clusters database. Examples of global enqueues include the database mount lock, used to control which instances can concurrently mount the database, and library cache locks, used to signal object definition changes that might invalidate library cache objects. In a noncluster database, such locks are local to the instance. The Global Enqueue Service Monitor (LMON) is responsible for monitoring the entire cluster to manage global enqueues and resources. The process ensures that the LMD processes (discussed on the next page) and the storage they require for global enqueue information are functioning correctly. LMON also manages instance and process expirations and the recovery processing associated with cluster enqueues.



Global Enqueue Service Daemon

• Manages access to global enqueues and resources

• Handles resource requests from other instances

InstanceLMS0LMON

LCK

LMD

DIAG

OSDClusterware

Global Enqueue Service DaemonThe current status of each global enqueue is maintained in a shared memory area of one of the active instances. The status indicates which instances, if any, have the right to use that resource. The Global Enqueue Service Daemon (LMD) is the process responsible for managing requests for global enqueues and updating the status of enqueues when they are granted to, or revoked from, an instance. The LMD process for the instance holding a specific enqueue record receives requests for that resource from the LMD process of an instance requesting that resource (the remote instance). If the resource is available, the LMD process updates the enqueue status and notifies the LMD process on the remote instance. If the enqueue is in use by another instance and not currently available, the LMD process sends its own request to the LMD process on the holding instance for the enqueue to be released. When the enqueue becomes available, the original LMD process coordinates the activity, changing the status in its memory area and informing the remote instance of its availability.The LMD process also manages the detection of deadlocks that can occur if enqueue requests are made by different instances for two or more enqueues.



Global Cache Service Processes

• Manage access to data blocks across the cluster

• Ship block images between the buffer caches of different instances (Cache Fusion)

InstanceLMS0LMON

LCK

LMD

DIAG

OSDClusterware

Global Cache Service ProcessesThe Global Cache Service Processes (LMSn) manage requests for data access across the cluster. They ensure that images of the same block can only appear in the buffer caches of two different instances if the block contents are valid for each instance. LMSn processes coordinate block access by sending messages between an instance requesting access to a specific block to an instance holding an image of that block, if there is one. An LMSn process on the holding instance can:

• Build a read consistent block image and ship it to the buffer cache of the requesting instance, or

• Forward the current block image to the requesting instance.The LMSn processes coordinate update activity on blocks, allowing only one instance at a time to make changes and ensuring that those changes are made to the most current version of the block. While an instance maintains a block image in its buffer cache, whether read from disk or transferred from another instance, LMSn processes are responsible for maintaining a record of that image, including updating a status flag if the block changes to and from an updateable mode. Real Application Clusters software provides for up to ten LMSn processes. The number of LMSn processes varies depending on the amount of messaging between nodes in the cluster.



LCK Process

• Assists LMSn processes with requests for resources between instances

• Handles cache resource request that do not involve Cache Fusion: dictionary and row cache

InstanceLMS0LMON

LCK

LMD

DIAG

OSDClusterware

LCK ProcessThe LCK process manages instance resource requests and cross-instance call operations. These are calls associated with coordinating access to dictionary and row cache objects. These cache resources, and hence the LCK processes on the instances, do not involve Cache Fusion block transfers between the instances.



Diagnosability Daemon

• Captures diagnostic information related to instance process failures

• Logs information to the directory specified by BACKGROUND_DUMP_DEST

• Process cannot be configured and should not be stopped

InstanceLMS0LMON

LCK

LMD

DIAG

OSDClusterware

Diagnosability DaemonThe Diagnosability Daemon (DIAG) captures diagnostic information related to process and instance failures. This information can be used Oracle World Wide Support to help analyze and resolve problems with your database and instances.The DIAG process writes its diagnostic information to files in a subdirectory of the directory specified by the initialization parameter BACKGROUND_DUMP_DEST. The subdirectories are named cdmp_timestamp, where timestamp identifies when the subdirectory, and trace information, was written. The following is an example taken from a Solaris system:

$ ls cdmp*cdmp_20011218004549:p1n1_arc0_21550.trw p1n1_lmon_21528.trw …p1n1_lmd0_21530.trw p1n1_p001_21565.trw

The command was issued from the BACKGROUND_DUMP_DEST directory and shows DIAG output created at 45 minutes and 49 seconds after midnight on December 18, 2001. The DIAG process starts automatically, is not tunable, and should not be disabled or removed. It can be automatically restarted by other background processes if this becomes necessary.



Practice Overview: Identifying Components of Real Application Clusters

This practice covers the following topics:• Identifying files belonging to a Real Application

Clusters database• Identifying Real Application Clusters background

processes



Summary

In this lesson, you should have learned how to:• Confirm that a cluster contains the general

(platform-independent) components to support Real Application Clusters

• Confirm that the components exist to run a Real Application Clusters instance

• Locate Real Application Clusters files and background processes


Installation and Conversion



Objectives

After completing this lesson, you should be able to do the following:• Perform preinstallation steps• Select installation options related to RAC• Migrate from an Oracle Parallel Server database• Modify a single instance database to a Real

Application Clusters database



Preliminary Cluster Preparations

• Build cluster using interconnect

• Install and configure OSD clusterware– This software is

platform specific– Installation and

configuration complexity varies by system

OSDClusterware

OSDClusterware

Interconnect

Preliminary Cluster PreparationsThe Oracle9i Enterprise Edition, which is required for Real Application Clusters, provides single instance database software and the optional software to operate Real Application Clusters databases. To install the optional Real Application Clusters software, you must perform the installation on an active cluster. This is a cluster where you have configured an interconnect between the node and an active OSD layer on each node.For a Windows cluster, your Oracle installation material will include an Oracle Cluster Setup Wizard that you install prior to running Oracle Universal Installer (OUI). Once you have setup the necessary logical partitions (described in the Oracle9i Database Installation Guide for Windows manual), this wizard walks you through the OSD installation andconfiguration steps. For other systems, the vendor will provide your cluster software and the related installation and configuration information.You should also confirm that you have access to your administrative account on all of the nodes in the cluster from the node where you intend to run the OUI.



Preliminary Cluster Preparations

• Ensure all nodes have access to shared disks

• Create any required file systems or partitions– Determine number and

sizes based on your circumstances

– Limit yourself to a few distinct sizes, particularly for spare partitions

OSDClusterware

OSDClusterware

Shared disksand partitions

Preliminary Cluster PreparationsBefore starting the installation of the Oracle9i Enterprise Edition Real Application Clusters software, you should determine what shared files you will need and create the file systems or raw partitions to support them. You should also confirm that the Oracle user has read and write access to these files or partitions from all the nodes in the cluster where you intend to run your Real Application Clusters database.The file systems or partitions you will need depend on a number of factors:

• Are you migrating an existing Oracle Parallel Server database?• Will you create the default database defined by the Database Configuration Assistant

(DBCA) while running Oracle Universal Installer?• Do you plan to create a database manually?• Do you wish to create spare partitions for use in emergencies such as restoring a lost file

to a different device or adding a data file due to space problems? The shared files required for a default installation and database are defined later. Your own circumstances determine if you want additional or a different allocation than these.If you are using raw partitions, you may find it easier to create partitions with a limited number of distinct sizes, preferably no more than five or six, particularly for spare partitions that you may need later. This simplifies partition management for system administrators.



Number

12112 / instance2211111 / cluster

Cluster Files or Partitions forDefault Database

Minimum Size

400 MB 312 MB 160 MB 120 MB120 MB 110 MB100 MB 90 MB 70 MB 12 MB 5 MB100 MB

File or Partition Uses

System tablespaceUndo tablespaces Sample schemas tablespace Users tablespaceOnline redo log files Control files Temporary and OLAP tablespacesinterMedia Text tablespaceIndex tablespaceTools tablespaceServer parameter fileDatabase server configuration file

Cluster Files or Partitions for Default DatabaseOne of the OUI options allows you define characteristics of database you would like to have built. The DBCA creates scripts based on your input regarding the intended use of your database. The DBCA can run the script automatically as a final step of the installation process, or DBCA can save the script for you to execute manually at a later time. The table shows the files or partitions you will need to support a two-instance database built with the DBCA preconfigured options, including the sizes of the related files. If you decide to use rollback segments rather than automatic undo management, you will need a single rollback tablespace with at least 625 MB of available space instead of the two undo tablespaces. Also, you can elect not to use a server parameter file, in which case you will not need the 5 MB partition shown in the table.In the cases of redo log files and tablespace data files, the sizes shown are taken from the commands in the DBCA script to build a preconfigured database. If you are using raw partitions, the partitions must be sufficiently large to accommodate these file sizes plus any space required by header information written by some operating systems into each partition. For simplicity, add 1 MB to the sizes shown in the table for the redo and data partitions. If you are using a few, standard partition sizes as recommended earlier, you need to select, for each partition, the smallest one that is larger than the size shown in the table.



Global Services Daemon

• The GSD configuration file must be identified to complete a successful installation of Real Application Clusters

• The GSD must be running if you create a database using the DBCA (the DBCA help text contains instructions on how to manage the GSD)

• If you build your database manually, or migrate from an Oracle8i to an Oracle9i database, you will need to configure and start GSD manually on each node to use tools other than SQL*Plus

Global Services DaemonWhen you install the Oracle Real Application Clusters option with the OUI, you are prompted for the location of the shared file where the configuration information for your databases will be stored. This location will be accessed by the GSD and shared by all of your Real Application Clusters databases. You can initialize the contents of this configuration file by executing the srvconfig –init command . If you allow the OUI to build a Real Application Clusters database for you as part of the installation process, the information about this database will be added to the configuration file. The GSD will also be started as part of the process.If you run the DBCA independently of the OUI, you will need to start the GSD (as described in the next lesson) before starting DBCA. You do not need the GSD if you build and manage your database with commands entered through the SQL*Plus tool.Note: If you are migrating from an Oracle8i Parallel Server database, you can use the srvconfig –conv command to convert your database configuration file, db_name.conf, to a shared raw device for the GSD in your Oracle9i Real Application Clusters database.



Information Repository for the Database Server Configuration

These steps are an example from a cluster file system.1. Create a new file somewhere on the cluster file

system, for example: /var/opt/oracle/rac_gsdconf

2. Create the file /var/opt/oracle/srvConfig.loc3. Add the following line to this file:

srvconfig_loc=/var/opt/oracle/rac_gsdconf

4. Initialize the GSD configuration file5. Add your database name with the SRVCTL utility:

srvctl add db -p db_name -o oracle_home

6. Add each instance name with the SRVCTL utility: srvctl add instance -p db_name -i sid -n node

Information Repository for the Database Server ConfigurationTo use many database management tools, you need to create a database management configuration file. The example on the slide shows the steps you use on a system using a cluster file system. The steps below are for a system using raw partitions. They are based on a Solaris system. Your GSD configuration will depend on your own system environment.

1. Create a raw partition, /dev/vx/rdsk/rac9i/rac_gsdconf, of at least 100 MB

2. Create a file, called srvConfig.loc, in the directory named /var/opt/oracle3. Add the following entry in the srvConfig.loc file:

srvconfig_loc=/dev/vx/rdsk/rac9i/rac_gsdconf

This entry points to the partition created in step 1, where the configuration informationwill be stored.

4. Initialize the configuration with the srvconfig -init command as described previously in this lesson

5. Execute the SRVCTL add db command to identify your database:srvctl add db -p db_name -o oracle_home

6. For each instance, execute the SRVCTL add instance command:srvctl add instance -p db_name -i sid -n node



Installation Options

Identify a node where you will run OUI to install Oracle9i Enterprise Edition with the Real Application Clusters option

Identify additional nodes where OUI will install Oracle9i Enterprise Edition with the Real Application Clusters option

Define the type of database to be created by DBCA: preconfigured, custom, or none

Installation OptionsWith a correctly configured cluster (see your platform specific documentation for details), you should only need to run the OUI on one of your nodes. The OUI will perform the necessary operations to complete the Real Application Clusters installation on each of the nodes that you select. During the installation, you will need to select options that are necessary for a successful Real Application Clusters installation. Some of the key options you must select include the following:

• Oracle9i as the product to be installed• Enterprise Edition as the type of installation• An option to determine what steps the DBCA will execute as part of the installation

processing- Build one of the preconfigured databases (data warehouse, transaction processing, or

general purpose); the best option if you are unfamiliar with cluster databases- Prompt you to define a customized database configuration; appropriate if you have

already configured the shared disk structures for all of the database files- Perform no database creation steps (which you select with the Software Only

installation option); a valid option if you are migrating an existing cluster database from an earlier release

• List of nodes where the Oracle software will be installed.



OUI Database Configuration Types

Software and options installed

Define own structures

Database built

Oracle Net configured

Listeners started

Instances started

PreconfiguredDatabase

Yes

No

Yes

Yes

Yes

Yes

Customized Database

Yes

Yes

Yes

Yes

Yes

Yes

Software Only

Yes

Yes

No

No

No

No

OUI Database Configuration TypesThe preconfigured database and customized database options enable the OUI, working with the DBCA, to not just install the software, including the Real Application Clusters and other selected options, but to perform much of the configuration required for your database environment. This includes building and executing database creation scripts, configuring the Oracle Net files, and starting the listeners and instances on each of your nodes. A software only installation does not invoke the DBCA and only installs the selected software without making any attempt to configure or start database or Oracle Net processes.Although you can define exactly what tablespaces, files, and options you wish to include when you select the customized and software only options, the preconfigured database option does offer some configuration options which affect the initialization parameter values applied to the instances:

• Data warehouse, for query-intensive databases• Transaction processing, for transaction-intensive databases• General purpose, for hybrid databases with a mixture of query-based and transaction-

based applications.



Additional Installation Considerations

• Address cluster configuration problems if the OUI fails to identify additional nodes when installing a preconfigured or custom database

• Bypass the OUI Upgrading or Migrating an Existing Database page

• Name a raw device or cluster file system location in response to requests for file names by the OUI

• Select the option labeled Create persistent initialization parameters file (spfile) to configure a server parameter file (recommended)

• Run OUI on all nodes, if necessary, for software only installations

Additional Installation ConsiderationsAfter you select one of the options to build a preconfigured database (Data Warehouse, Transaction Processing, or General Purpose) or to build a customized database (New Database), the OUI should present you with a list of available nodes. You select the nodes, in addition to the local node where you are running OUI, where you want the software installed and configured. If the node selection page does not appear, your cluster setup may be flawed. You may need to rerun OUI after identifying and fixing the problem.You should not attempt to use the option to upgrade or migrate an existing database, even if you intend to convert it to Oracle9i. The migration tool used by this option is not able to handle Real Application Clusters databases.Remember to provide the name of a raw device or cluster file system location for any file names you need to provide to OUI or the DBCA.Oracle encourages you to manage initialization parameters for your Real Application Clusters database with a server parameter file (SPFILE). Check the selection box labeled Create persistent initialization parameters file (spfile)to enable OUI to create the SPFILE for you in a predefined shared disk location. You may need to run OUI on all of your nodes if you perform a software only installation. See your platform-specific documentation for details.



Migrate to Oracle9i Real Application Clusters

Simple to migrate? (1 easiest, 3 hardest)Migrate subset of database?Migrate Parallel Serverdatabase?Migration time independenton database size?Requires SRVCONFIG configuration data?

Migration Assistant

1

NoNo

Yes

Yes

Migration Utility

2

NoYes

Yes

Yes

Export/ Import

3

YesYes

No

Yes

Migrate to Oracle9i Real Application ClustersIn addition to the normal migration steps required for a nonclustered database, you will need to provide database configuration information for your Oracle9i installation. You must do this after you complete your Oracle9i Real Application Clusters installation but before you migrate your database. Check your platform-specific documentation for instructions on how to perform this step on your cluster. On UNIX, for example, you would execute the following command:srvconfig –conv $ORACLE_HOME/ops/db_name.conf

where db_name.conf is the name of your Oracle8i OPSCONF configuration file.You can use any of the migration options to convert from an earlier Oracle release to an Oracle9i Real Application Clusters database:

• The Migration Utility is the preferred method requiring minimal resources and able to migrate very large databases faster than other methods.

• The Data Migration Assistant provides a simple interface to the Migration Utility. Note that you cannot use the Data Migration Assistant for an Oracle Parallel Server database.

• Export/Import is slower and requires more work than the Migration Utility. However, choose this option for migrating only part of your database or for converting a nonclustered database. You may also combine data copying methods with Export/Import.



Convert a Single Instance to Real Application Clusters

Task ID123 *4 *567 *8 *910

Task DescriptionConfigure hardwareEvaluate tablespaces and log filesCreate shared file system or raw devices Extract data from old databaseInstall operating system-dependent cluster softwareInstall Real Application Clusters optionCreate the databaseLoad data from old database into new databaseAdjust parametersStart the database

* These steps may not be required if currently using shared files

Convert a Single Instance to Real Application ClustersTo modify a single instance to Real Application Clusters, complete the following tasks:

• Task 1: Configure hardware• Task 2: Evaluate tablespaces and log files currently used in your single instance database• Task 3: Create a shared file system or raw devices, if necessary, to relocate your

database to a shared file environment• Task 4: Copy files or data from old database

Note: If your current database is already stored on the system where you will be running your Real Application Clusters database, you can copy the database files to the new, shared location and skip step 4 as well as steps 7 and 8. You can use RMAN or operating system commands to make the file copies, or use RMAN or Export for copying database contents.

• Task 5: Install operating system-dependent cluster software• Task 6: Install Oracle9i Enterprise Edition and the Oracle9i Real Application Clusters

option• Task 7: Create the database• Task 8: Load data from old database into new database with RMAN or Import• Task 9: Adjust parameters• Task 10: Start the database



Configure Hardware

Prepare cluster hardware as discussed earlier• Install and test the

cluster interconnect• Ensure all nodes have

access to shared disks

OSDClusterware

OSDClusterware

Shared disks

Task 1: Configure Hardware You need to enable your cluster to support Oracle9i Enterprise Edition Real Application Clusters software just as you would for an initial installation. This includes the provision of a cluster interconnect and access to a shared disk array.



Evaluate Tablespace Requirements

• Evaluate the tablespaces and related data files in your current database– Disable automatic extension on all data and temp files

if using raw partitions– Remove any unwanted files to reduce conversion

effort• Plan your rollback strategy

– If you employ automatic undo management, you will need an undo tablespace for each instance

– If you employ traditional rollback segments, you will need sufficient space for at least one segment for every instance

Task 2: Evaluate Tablespace RequirementsUnless you are using a cluster file system, you must prevent database files attempting to extend beyond a raw partition boundary. Identify and modify all data and temp files with automatic extension enabled that will be on raw devices using commands similar to these:SELECT file_name, autoextensibleFROM dba_data_files;ALTER DATABASE DATAFILE '/d2/ora/indx1.dbf' AUTOEXTEND OFF;

If you are not using a cluster file system, each file (data, online redo log group member, and copy of the control file) in your Real Application Clusters database will have to be stored on a shared device. If your database contains a wide variety of file sizes, you may want to reassign your database contents. The goal is to restrict the number of distinct partition sizes needed for your database to simplify storage management. If you need to relocate your Real Application Clusters database to a shared file system, you should remove any files that will not be needed to avoid the effort of moving them.If you employ automatic undo management in your Real Application Clusters database, you need a separate undo tablespace for each instance. On the other hand, if you use traditional rollback segments, you need at least one segment for each instance (although more will probably be required to avoid performance bottlenecks). You should decide your rollback strategy in order to create appropriate partitions in the next task.



Evaluate Log File Requirements

Plan your online redo log requirements.• Each Real Application Clusters instance will need

its own thread of redo• Each redo thread requires at least two redo log

groups• Instances may have different numbers and sizes of

redo log groups if they support unequal processing activity

• Each redo group should have at least two members• Log groups on different instances may have

different numbers of members, although this is not a common configuration

Task 2: Evaluate Log File RequirementsYour Real Application Clusters database needs a set of log files, known as a thread, for each instance. To plan your shared storage, determine if you need additional online redo log files to support your extra instances or if you have sufficient log files currently. If you wish, each thread can contain a different number of redo log groups and the groups can be based on different log file sizes. This can be appropriate for databases where you expect to partition the workload across the instances such that some instances will generate more redo than others. However, if you have no specific need to assign dissimilar work to different instances, you should configure your instances identically, including the numbers and sizes of redo files allocated to a thread. This makes it simpler to employ load balancing or fail over initially or at a future time.As in a nonclustered database, you may have different redo log groups with different numbers of members. However, this approach has little benefit. Note: You can rely on operating system or disk mirroring to provide multiple copies of your online redo log files rather than creating groups with more than one member.



Create Shared File System or Raw Devices

• Shared files are needed for the cluster configuration data and for each– Copy of the control file– Data file– Online redo log group

member (across all threads)• A shared location for the

SPFILE is required to configure a server parameter file

Shared disk partitions

Task 3: Create Shared File System or Raw Devices Use the size of your current control file as a guide for how large the partitions should be for the copies of this file in your Real Application Clusters database. Be aware that the control file may be larger when it has more redo threads to track and needs to record additional archive log history. Also allow for any planned growth in your database and the number of files that will be required to support this growth. You should consider making the partitions for the control file copies at least 110 MB or the current size plus 1 MB, whichever is larger.Base the partition sizes for your data, temp, and online redo log files on their current sizes, remembering to add sufficient space for any required system overhead in each partition. Also, your partitions need to reflect any adjustments you made, or plan to make, to the tablespaces or redo log allocation (to provide automatic undo management tablespaces, for example), based on your evaluation in the previous task.If you want to create your database with a server parameter file (SPFILE), you will also need a partition to hold a shared copy of this file. Make a partition which is at least 5 MB but, if you are restricting your partitions to a few fixed sizes, choose the smallest size that is greater than 5 MB.Note: This step is only needed if you are migrating your database to raw devices. If your system that supports clustered files, ensure you have space to support the database files.



Extract Data from Old Database

There are a number of ways to extract the data from your old database• RMAN is the easiest, particularly if you are using

raw partitions• The Export utility is also an option• You may also use unload routines if you have them

available

Task 4: Extract Data from Old DatabaseThe easiest and fastest way to move your database contents from a nonshared file system to your cluster storage is with Recovery Manager (RMAN). RMAN is able to handle different file types, including raw partitions, and so can restore files copied from a standard file system to a cluster file system or to raw partitions transparently.You can also use the Export utility to perform a logical extraction of your database contents. However, once exported, it can be a slow process to import the data, even into a Real Application Clusters database. Export is useful, however, if you only wish to migrate part of your database to your Real Application Clusters system. In such cases, you can perform user or table level exports to obtain only the data you need. The time needed for the import will depend on how much data your Export dump file contains.You can also use other methods to migrate your data, such as an unload/reload utility, from your single instance to your Real Application Clusters database. Such utilities, if you have them available, may allow you to be selective in what data you migrate. They may even allow you to modify the segment definitions or data before loading them into your Real Application Clusters database.



Installation Steps

• Install the OSD clusterware as described earlier in this lesson

• Run the OUI and select the Real Application Clusters option

• Select the New Database option and define prepared database structure

Tasks 5 and 6: Install OSD and Real Application ClustersThe earlier sections of this lesson contain an overview of installation of the OSD cluster software. Detailed instructions are available in your operating system-dependent documentation for clustered databases.Similarly, the installation of the Oracle9i Real Application Clusters software (using the OUI), is covered earlier in this lesson. If you already have Oracle9i installed, you have to run OUI to add the Real Application Clusters option after your OSD clusterware is operational.To build your database with the DBCA, select the New Database option rather than one of the preconfigured database options. Supply information based on the database configuration you defined in task 2. If you want to build a preconfigured sample database during your initial installation of Real Application Clusters, you may do this. Later, you can run DBCA and define your production database structure using the New Database option.



Create the Database

• The preferred method is to use DBCA

• You can execute your own commands or scripts in a SQL*Plus session


Instance

OSDClusterware

Databasefiles

Task 7: Create the DatabaseYou may already have created your database by allowing DBCA to build and execute the required scripts for you as part of the installation with OUI. You can run DBCA anytime after your installation to build your database according to the requirements you have already defined.If you desire, you can build your database without the help of the DBCA. To do this, you will need to connect to a SQL*Plus session with administrator privileges and start an instance with the STARTUP NOMOUNT command. You will need to execute the CREATE DATABASE command plus the commands add your required tablespaces, redo log threads, and so on. You can either type the necessary commands directly at the SQL*Plus prompts or execute a script, prepared manually or derived from a DBCA script.



Create the Database

CREATE DATABASE . . .MAXINSTANCES 2 MAXLOGFILES 16 MAXLOGHISTORY 100. . .UNDO TABLESPACE undo01. . .

ALTER DATABASE ADD LOGFILE THREAD 2. . .

ALTER DATABASE ENABLE THREAD 2

CREATE UNDO TABLESPACE undo02 . . .

Task 7: Create the Database (continued)If you choose to build the database yourself, rather than using DBCA which is the recommended approach, you may need to override the default values for a number of CREATE DATABASE options:•MAXINSTANCES: Set to at least the number of instances you expect to run concurrently

(allowing for possible expansion to more instances in the future)•MAXLOGFILES: Set to the number of redo log groups you plan to define•MAXLOGHISTORY: Set to the maximum number of redo logs you expect all of your

instances to generate between database backups.Each instance you run concurrently requires access to its own undo data. If you are using automatic undo, you will need an undo tablespace for each instance, which you identify with the UNDO_TABLESPACE parameter value for each instance. If you persist in using traditional rollback segments, you will need at least one for each instance. Although you can use public rollback segments, management is easier if you make them private and name the ones you want active in an instance with the ROLLBACK_SEGMENT parameter.You will also require an enabled thread of redo for each instance. Although you can enable redo threads publicly, you are advised to use private threads and identify them by the value in the THREAD parameter for each instance.



Load Data into New Database

imp system/manager file=upgrade.dmp full=y log=upgrade.log

Instance

OSDClusterware

Instance

OSDClusterware

Export dump file Real ApplicationClustersdatabaseImport utility

Task 8: Load Data into New DatabaseThis step, like step 4, is only needed if you are relocating your database in order to use a cluster file system or raw partitions to support the files. You need to recreate your database contents based on the method you used in step 4 to extract the information from your non-shared database files.If you used RMAN, the preferred method, simply restore the files to the new location. RMAN will handle any issues with file or partition overhead for you automatically, regardless of the original file structure.The example is based on the use of an export command in task 4. If you decide to use the Export/Import utilities, you will need to modify this sample command to meet your specific requirements for file names, best performance, and so on. If you completed the previous steps successfully, the Import utility should complete without errors. You may encounter problems if you have not correctly built the same tablespaces in your Real Application Clusters database as were in the database you exported. In such cases, the Import utility will attempt to build the tablespaces using the original, non-shared file system. Even if this succeeds, these data files will not be visible across the cluster. You will have to drop and recreate the tablespaces before you can use multiple instances and then re-import the data.If you use some other method to load your data, you must adhere to its requirements.



Adjust Parameters

thread = 1instance_number = 1undo_tablespace = undo01#rollback_segments = rb1instance_name = PRN1

Instance-specific parameters

Common parameterscluster_database = TRUEdb_block_size = 8192#db_cache_size = 10000#value reduced by ceil(350*10000/(350+8192))db_cache_size = 9590

Task 9: Adjust ParametersTo start multiple instances against your database, you will need to define certain individual initialization parameter values specific for each instance. If you have implemented a server parameter file during your installation, these parameters should already be set correctly. If you are using text parameter files, your are encouraged to convert to a server parameter file. While continuing to employ text initialization parameter files, you will need to maintain one file for each instance and one with parameter values common to all of your instances. Use the IFILE parameter to import the common file into each instance-specific parameter file. Parameters that must be given different values for each instance include THREAD, INSTANCE_NUMBER, either UNDO_TABLESPACE or ROLLBACK_SEGMENTS, and, if used, INSTANCE_NAME. You will also need to CLUSTER_DATABASE=TRUE in the common initialization file in order to open more than one instance concurrently.To maintain your current overall SGA size, you will need to modify the parameters controlling buffer cache sizes for standard and nonstandard block sizes to allow for additional buffer space required by a Real Application Clusters instance. To maintain the equivalent size of your nonclustered database SGA, reduce the number of buffers by CEIL(350 * blockcount /(350 + blocksize)), where blockcount is the number of buffers in the original cache and blocksize is the size of the cache buffers.



Start the Database

$ sqlplus /nolog

SQL*Plus: Release 9.0.1.0.0 – Production

(c) Copyright 2001 Oracle Corporation. All rights reserved.

SQL> CONNECT sys/oracle as sysdbaConnected.SQL> STARTUP PFILE=/mytest/pfiles/initPRN1.ora

# Parameter File initPRN1.oraspfile = '/dev/vx/rdsk/opsdg2/rac_pfile_PR'

Task 10: Start the DatabaseYou can start your individual instances just as you would start any other instance:

• Connect to the server and start a SQL*Plus session in privileged mode• Issue a STARTUP command, including the PFILE parameter if you are not using a

default name and location• Repeat these steps on each node in the cluster to start your entire cluster database.

There are other tools you can use to manage your Real Application Clusters environment, including instance starting up and shutting down individual or multiple instances. These are described later in this course. To start your instances manually with a server parameter file, you may need to use a text parameter file containing just an SPFILE entry. The value for this parameter would be the location of, or the name of a link to, the shared partition where the server parameter file is located. For example:spfile = '/dev/vx/rdsk/opsdg2/rac_pfile_PR'



Additional Migration Considerations

• Have a current backup available before migrating• Include a thread number variable (%T or %t) in your LOG_ARCHIVE_FORMAT initialization parameter

• Consider a second, shared location for the archive files that all instances can access during recovery

• Skip tasks 3, 4, 7, and 8 during a conversion from a nonclustered database already on a shared file system

• Prepare to adjust additional tuning parameters as you monitor the performance of your new instances

• Include option to retry startup commands in scripts

Additional Migration ConsiderationsDuring your migration from a nonclustered database to Real Application Clusters, you should be cognizant of the following issues:

• You should have a backup and recovery strategy in place and a current backup available before attempting your migration

• If you are archiving, your archive log file names should include a thread number; you may want to adjust the LOG_ARCHIVE_FORMAT initialization parameter to accommodate this (the %T or %t variables define the thread number's location in the format string)

• Consider defining a second, optional location for the archive files (on a mounted or shared drive shared drive, not raw devices, if you are not using a cluster file system) so that all instances can access the same location for recovery operations

• If you are migrating from a database that is already defined on a shared file system, you can skip tasks 3, 4, 7, and 8 during a conversion from a nonclustered to Real Application Clusters database

• You may need to make parameter value changes, in addition to those discussed in task 9, as you begin to tune your Real Application Clusters instances

• Consider adding the RETRY option to the STARTUP command in your instance startup scripts to avoid failures that can occur if an active instance is performing recovery.



Practice Overview: Modifying a Database to Support a Second Instance

This practice covers the following topics:• Modifying or creating initialization parameter files• Adding second redo thread• Adding second rollback segment• Starting instances with appropriate parameters• Examining instance information with dynamic

performance tables



Summary

In this lesson, you should have learned how to:• Prepare a cluster for a Real Application Clusters

installation• Define the required shared disk resources for a

clustered database• Migrate to a Real Application Clusters database

from an earlier release or a nonclustered Oracle9idatabase


Management and Configuration Tools



Objectives

After completing this lesson, you should be able to do the following:• Manage the Global Services Daemon (GSD)• Use tools to configure and manage Real Application

Clusters– Server Control Utility– Oracle Enterprise Manager (OEM)

• Create and manage initialization parameter files for cluster databases– Client-side (PFILE) and include files (IFILE)– Server parameter file (SPFILE)



GSD Management

• Start a GSD on all the nodes in your Real Applications Clusters database for use by the management tools including– SRVCTL– the DBCA– OEM

• Run only one GSD on each node

gsdservice -install

gsdservice -start

gsdservice -remove

gsd

Successfully started the daemon on the local node.

UNIX

Windows

GSD ManagementClients of GSD, such as SRVCTL, the DBCA, and Oracle Enterprise Manager, interact with the daemon to perform various manageability operations on the nodes in your cluster. You must start the GSD on all the nodes in your Real Applications Clusters database before you issue SRVCTL commands or attempt to employ the other tools across the cluster. However, you need only one GSD on each node no matter how many cluster databases you create.On Windows platforms, you use the gsdservice commands to manage the GSD service, OracleGSD. The examples show the options to install, start, and stop and remove the service: -install, -start, and -remove respectively. The GSD service is located in the %ORACLE_HOME%\bin directory and records activity in the %ORACLE_HOME%\srvm\log\gsdservice.log file. On UNIX, the name of the daemon is gsd and the daemon is located in the $ORACLE_HOME/bin directory. You start the daemon with the gsd command as shown in the example. Logging information is written to the file $ORACLE_HOME/srvm/log/gsdaemon.log.In an earlier lesson, you saw the results of attempting to start a duplicate GSD on a UNIX node.



Server Control Utility

The Server Control Utility (SRVCTL) • Is the preferred tool to

administer your Real Application Clusters database environment

• Manages cluster database configuration information used by several Oracle tools

• Provides cluster database management commands

• Requires the GSD to be runningNode 2

GSD

SRVCTL

Agent

Node 1

GSD

SRVCTL

Agent

Server Control Utility (SRVCTL)Oracle Corporation recommends that you use the Server Control Utility, SRVCTL, to administer your Real Application Clusters database environment. SRVCTL manages the configuration information required for a cluster-wide perspective of your database and its instances. This information is used by SRVCTL database management commands as well as by several other tools. For example, node and instance mappings needed for discovery and monitoring operations performed by Oracle Enterprise Manager (OEM) and its intelligent agents is generated by SRVCTL. Many of these tools execute SRVCTL commands to complete the operations requested through their graphical user interface (GUI).SRVCTL works with the GSD to manage and retrieve cluster and database configuration information stored in the shared disk location discussed in a previous lesson.



SRVCTL Command Syntax

srvctl

Usage: srvctl {start|stop|status|config|add|delete|rename|move|get|set|unset} [options]

srvctl config -h

Usage: srvctl config [-p <db_name> [-n <node>]][-D <dbglvl>] [-h] [-V]

-p <db_name> Show configuration for specified cluster database . . .

-n <node> Only show services on named node-D <debuglvl> Debug level-h Print usage-V Show version

SRVCTL Command SyntaxTo see a list of available command options, enter:

srvctl

To see the online command syntax and options for each SRVCTL command, enter:srvctl command option -h

where command option is one of the valid options such as start, stop, or status.Note: To use SRVCTL, you must have already created the configuration information for the database that you want to administer by using either the DBCA or the srvctl addcommand as described in an earlier lesson.



SRVCTL Cluster DatabaseConfiguration Tasks

Modify database configuration information• Add and delete cluster databases • Add instances to and delete instances from a

cluster database• Rename instances • Move instances• Set and unset the environment for an entire cluster

database• Set and unset instance environments

SRVCTL Cluster Database Configuration TasksUse SRVCTL to update the cluster database configuration information repository stored on the shared file and used by GSD to execute commands appropriate for the cluster database or for specific instances. The following types of information can be created or modified with SRVCTL:

• Define a new cluster database configuration or remove obsolete database configuration information

• Add information about a new instance to a cluster database configuration or remove instance information from a cluster database

• Rename instance name within a cluster database configuration• Change the node where an instance will run in a cluster database configuration• Set and unset the definitions used to assign environment variables for an entire cluster

database• Set and unset the definitions used to assign environment variables for an instance in a

cluster database configurationThe specific commands to accomplish these tasks are covered in the following pages.



Add and Delete Databases

Add database configuration information

Remove database configuration information

srvctl add db –p T2 –o $ORACLE_HOME

srvctl delete db –p T1

Add and Delete Databases The srvctl add db command creates the configuration information for the Real Application Clusters database. The following syntax adds the configuration information for a Real Application Clusters database to the configuration repository:

srvctl add db -p db_name -o oracle_home

This database is identified by the name you provide for db_name and the oracle_homevalue must be the location where you installed Oracle9i Real Application clusters. The example shows the creation of the database T2 on a UNIX system. You must execute the srvctl add instance command, described on the next page, to add instance configurations to the database.Use the srvctl delete db command to delete the static configuration for a Real Application Clusters database. The following syntax deletes the Real Application Clusters database identified by the name that you provide:

srvctl delete db -p db_name

The example shows the removal of repository information for the database called T1.Note: Oracle Corporation recommends that you use the Instance Management feature of the DBCA to add and delete cluster databases and instances themselves.



Add and Delete Instances

Add instance configuration information

Remove instance configuration information

srvctl add instance –p T2 –i T2N3 –n raclust3

srvctl delete instance –p T2 –i T2N1

Add and Delete InstancesUse the srvctl add instance command to add static configuration information for an instance. This command only updates the configuration information in the repository; it does not create the database or the instance. The following syntax adds an instance, named instance_name, to the specified database on the node you identify with node_name:

srvctl add instance -p db_name -i instance_name -n node_name

The example adds the instance T2N3 on node RACLUST3 to the configuration information for database T2.The srvctl delete instance command deletes static configuration information for a Real Application Clusters instance. Use the following syntax to delete the configuration for the instance identified by the database name that you provide:

srvctl delete instance -p db_name -i instance_name

The example removes the instance T2N1 from the configuration information for database T2.Note: Oracle Corporation recommends that you use the Instance Management feature of the DBCA to add and delete cluster databases and instances.



Move and Rename Instances

Move instance to new node in the static configuration

Rename instance in the repository

srvctl move instance -p T2 -i T2N2 -n raclust1

srvctl rename instance -p T2 -i T2N2 -e T2N1

Move and Rename Instances Use the srvctl move instance command to move a Real Application Clusters instance from one node to another in the static configuration. The following syntax moves the named Real Application Clusters instance information, instance_name, belonging to the database defined by db_name, to the node identified by new_node:

srvctl move instance -p db_name -i instance_name -n new_node

To rename a Real Application Clusters database instance in the repository, execute the srvctl rename instance command. The command includes the database name, shown as db_name in the syntax below, and the old and new instance names, shown as old_name and new_name respectively. The full syntax for the command is:

srvctl rename instance -p db_name -i old_name -e new_name

In order to preserve a meaningful naming convention for your cluster database information, you will want to use these two commands together. First, move the configuration information for the renamed instance to the new location. Second, rename the instance in the configuration to include the ID of the new node in the instance name. The example shows such an operation, relocating instance T2N2 to the first node, RACLUST1, then renaming it T2N1 to reflect the “1” in the node name.



Store and Remove Environment Data

Store database-wide environment setting

Store instance-specific environment setting

Remove environment setting from single instance

srvctl set env -p T2 -t LANG=gb -i T2N1

srvctl set env -p T2 -t LANG=us

srvctl unset env -p T2 -t LANG -i T2N3

Store and Remove Environment DataTo store values in the repository that are used to define the environment when instances are started, use the srvctl set env command. You can set an environment value for an entire cluster database by naming the database in the following syntax, where name is the environment variable you want to set and value is what should be assigned to it:

srvctl set env -p db_name -t variable=value

You can also set the environment for a specific instance with the following syntax:srvctl set env -p db_name -t variable=value -i instance_name

Use the srvctl unset env command to remove, from the configuration data, the environment values that Oracle uses when you start instances from the repository. You can remove the environment value for an entire cluster database with the following syntax, naming the database and variable:

srvctl unset env -p db_name -t variable

You can also remove the environment data for specific instances from the repository by also including the instance name in the command, as shown in the following syntax:

srvctl unset env -p db_name -t variable -i instance_name

The first two examples set the language for database T2 to American and to English for its instance T2N1. The third example removes the language setting for instance T2N3.



SRVCTL Cluster Database Tasks

Manage cluster database components• Start cluster databases and instances• Stop cluster databases and instances• Start and stop listeners associated with a cluster

database• Start and stop listeners associated with a cluster

database instance• Obtain the status of a cluster database instance

SRVCTL Cluster Database TasksYou can execute the same SRVCTL cluster database commands to start and stop Oracle components that are invoked by the various tools that use SRVCTL. Most of the database commands in SRVCTL can be issued to work across the cluster or on individual nodes. These commands enable you to:

• Start and stop cluster databases• Start and stop cluster database instances• Start and stop listeners associated with a cluster database• Start and stop listeners associated with a cluster database instance• Obtain the status of a cluster database instance

The specific commands to accomplish these tasks are covered in the following pages. Note: Your database and instance information must be available in the configuration repository before you use SRVCTL to perform these operations. DBCA will store this information or you can provide it manually with the database configuration commands described on the preceding pages.



Starting Databases and Instances

Start all instances and listeners

Start two named instances with their listeners

Start listener only on one node

srvctl start -p T2 –i T2N1,T2N2

srvctl start -p T2

srvctl start –p T2 –n raclust3 –s lsnr

Starting Databases and InstancesThe key elements of the syntax of the SRVCTL command to start one, a subset of, or all instances in a Real Application Clusters database include:

srvctl start -p db_name [-i inst,...] [-n node,...][-s stage,...] [-x stage,...]

where-p db_name identifies the database against which the command is executed-i inst,... is the name of the instance, or a comma-separated list of instances, that are started (the default is all instances defined for db_name)-n node,... is the name of the node, or a comma-separated list of nodes, on which the instances are started (the default is all nodes with instances defined for db_name)-s stage is the type of component, either inst (for instance) or lsnr (for listener) that are started (the default is that both instances and listeners are started)-x stage, is the type of component, either inst (for instance) or lsnr (for listener) that are not started (the default is that neither instances and listeners are excluded)Note: You can include both stages with the –s and –x option: -s inst,lsnr.



Starting Databases and Instances

Start all instances with nondefault connect string

Start listeners and open instances with special parameter file on two nodes

srvctl start -p T2 –n raclust1,raclust2 \–o pfile=/ora/admin/initADMIN.ora

srvctl start -p T2 –c 'admin/adm1n as sysdba'

Starting Databases and Instances (continued)The additional syntax options for the srvctl start command are:

srvctl start ... [-c 'connstr'] [-o options] [-S level] [-D dbglvl] [-h]

where-c connstr defines the connect string for the startup operation (the default is: / as sysdba)-o options lists the startup command options, such as force, nomount, pfile= (with an appropriate path and parameter file name), and so on-S level defines the intermediate status level for the Console-D dbglvl sets the debug level-h displays the help information for the command or option



Stopping Databases and Instances

Stop an instance only, not the listener

Shut down instances and listeners on three nodes using a nondefault connect string

srvctl stop -p T2 \–n raclust2,raclust3,raclust4 \–c admin/adm1n as sysdba

srvctl stop –p T2 –i T2N3 –x lsnr

Stopping Databases and InstancesYou can shut down your database instance components with SRVCTL. The syntax and options for the srvctl stop command are similar to those for the srvctl start command, with options such as TRANSACTIONAL in place of MOUNT, and so on. The slide contains some typical examples of this command.



Inspect Status of Cluster Database

List the active instances

Show the status of two instances

Display the status of the listener on one node

srvctl status -p T2 –i T2N1,T2N4 –s inst

srvctl status -p T2

srvctl status –p T2 –n raclust2 –s lsnr

Inspect Status of Cluster DatabaseYou can use the srvctl status command to determine what components, such as instances and listeners, of your cluster database are running. The command uses the same syntax options as the srvctl start and srvctl stop commands. The examples on the slide show how SRVCTL commands can be used to show which instances are active in a cluster database and the status of the listener on one of the nodes. Below is sample output from the first of these examples:

$ srvctl status -p T2Instance is running on node: raclust1Instance is running on node: raclust2All of the listeners are running on node: raclust2All of the listeners are running on node: raclust1



Inspect Database Configuration Information

List the instances defined for a database

List environment information for a database

List environment information for an instance

srvctl get env -p T2

srvctl config -p T2

srvctl get env -p T2 –i T2N4

Inspect Database Configuration InformationYou can employ SRVCTL to examine the information in the cluster database configuration repository. The srvctl config command enables you to identify the existing Real Application Clusters databases. There are two formats for this command. The first includes no subcommands or options and lists all the cluster databases in your environment:

srvctl config

The second format, which includes the -p db_name syntax, lists the instances for the named database. The slide shows an example of this format. Use the srvctl get env command to obtain environment information for either an entire Real Application Clusters database or for a specific instance. The output from a command using the following syntax contains environment information for the entire Real Application Clusters database identified by the db_name value you provide:

srvctl get env -p db_name

A command with the following syntax displays environment information for a specific instance:

srvctl get env -p db_name -i instance_name



Enterprise Manager and Cluster Databases

Cluster databases

Enterprise Manager and Cluster DatabasesThe Oracle Enterprise Manager Console provides a central point to manage the Oracle environment through an intuitive graphical user interface (GUI). The Console can initiate a variety of cluster database management tasks with the Management Server component. From the Navigator pane, you can view and manage both single- and multiple-instance databases using essentially the same operations. Just as in single instance databases, cluster databases and all of their related elements can be administered using master/detail views and Navigator menus.After nodes are discovered, using the repository information added by DBCA or the SRVCTL utility, the Navigator tree displays cluster databases, their instances, and other related services, such as a listener. You can then use the Console to start, stop, and monitor services as well as to schedule jobs or register events, simultaneously performing these tasks on multiple nodes if you desire. You can also use the Console to manage schemas, security, and the storage features of cluster databases.Before using the Oracle Enterprise Manager Console, start the following components:

• An Oracle Intelligent Agent on each of the nodes• Management Server• Console



Displaying Objects in the Navigator Pane

Cluster instances

Displaying Objects in the Navigator PaneIn the Navigator tree, cluster databases are listed under the Databases folder as siblings to single-instance databases. Just as in single instance databases, each cluster database folder contains the instances and subfolders for Instance, Schema, Security, and Storage. By selecting objects within a Cluster Database subfolder, you can access property sheets to inspect and modify properties of these objects, just as you can for single-instance databases. All discovered instances will be displayed, under the Cluster Database Instances folder.With cluster databases, only the subfolders of the Instance folder are different from those of single instance databases. In the Instance folder, the instance database subfolders are split into two functional parts: Database-Specific File and Instance-Specific File Structures. The available database-specific functionality includes in-doubt transactions and resource consumer groups. All instance-specific functionality appears beneath the individual instance icons within the Cluster Database Instances subfolder and includes:

• Configuration and stored configuration information management• Session management• Lock information• Resource plan and resource plan schedule management



Starting a Cluster Database

Starting a Cluster DatabaseThe Console enables you to start an entire cluster database or selected instances within the cluster database. You can also select the required startup options, for example, MOUNT.The Cluster Database Startup/Shutdown Results dialog box automatically displays during a startup (or shutdown) operation. You can also initiate it with the following steps:

1. In the Navigator pane, expand Databases.2. Right-click on a cluster database.3. Select Results from the options menu that appears.

The display is updated dynamically as the operation progresses and graphically reflects the following states if the component is up and running (green flag) or if the component is down and not running (red flag).If the instances are started successfully, the Cluster Database Started message box appears with a successful message.



Stopping a Cluster Database

Stopping a Cluster DatabaseAs with the startup option available on the cluster database menu, you can choose to stop the entire cluster database or single instances. You can also select shutdown-specific options, such as IMMEDIATE. Only when all instances are shut down is the database considered closed. The Cluster Database Shutdown Progress dialog box displays the progress of the shutdown operation. Once the instances are shut down successfully, as shown in the slide, the Cluster Database Stopped message box also appears with a successful message. If the shutdown fails, the Cluster Database Stopped message box appears with a failure message.



Status Details Tab

Status Details TabIn addition to the Cluster Database Startup/Shutdown Results dialog box, you can also view the progress of startup and shutdown operations with the Status Details tab. While a startup or shutdown operation is running against a cluster, the Status Details tab progress display is shown and updated dynamically as the operation progresses.A successful shutdown operation for a two-node cluster looks like the Status Details tab shown in the slide.



Viewing Cluster Database Status

Viewing Cluster Database StatusThe Edit Cluster Database dialog box displays status information about the database, such as instances available in the cluster and the status of cluster components.To view status information about a database:

1. In the Navigator pane, expand Databases > database_name.2. Right-click on a cluster database under the Databases folder in the Navigator

pane.3. Select Edit from the options menu that appears.

The Edit Cluster Database dialog box appears and provides two views through the General Tab and the Status Details Tab.The General tab displays information about the currently running instances by querying V$ACTIVE_INSTANCES table. An example of the display, as shown on the slide, contains columns for the instance number, the instance name (in the format node:instance_name), and secondary (only populated if the node is an secondary instance in a primary/secondary instance configuration).Note: Because this dialog box requires a connection to a database, this dialog box will not appear if the cluster database is down.



Viewing Cluster Database Status

Viewing Cluster Database Status (continued)The Status Details Tab displays an overall view of the state of the cluster and related components, as shown in the slide. This tab displays the status of the various components, such as listeners and instances, for all nodes. The states of the components are indicated with the following graphical elements:

• Green flag: The component is up and running.• Red flag: The component is down and not running.• Timer: an operation is in progress and the Oracle Enterprise Manager cannot determine

the state of the component. This state occurs typically when the component startup or shutdown operation has not completed.

• Blank background: The component does not exist on this node or is not configured on the node.

Note: This tab is not available for a Windows cluster database because SRVCTL on Windows does not generate status details.



Job Management for a Cluster Database or Instance

Job Management for a Cluster Database or InstanceThe job scheduling system provides a highly reliable and flexible mechanism for you to schedule and automate repetitive jobs on both the cluster database and instances. The Console contains a full-featured tool that enables you to develop a customized schedule.You can create a job with a cluster database or instance as the destination. Use the tabs from the Create Job property sheet to specify the details of a new job. The main functions of each tab are as follows:

• General tab: Provide the job name, description, destination type (cluster database or instance), and destination

• Tasks tab: Choose the task(s) for the job• Parameters tab: Set the run-time parameters for the task or tasks you choose• Schedule tab: Schedule the time and frequency for Oracle Enterprise Manager to run the

job• Permissions tab: Specify the administrator to perform the job.



Registering Cluster Database Events

Registering Cluster Database EventsOracle Enterprise Manager includes an Event Management feature that enables you to monitor tests on cluster database instances. In addition to the tests available for single instance databases, there are specialized event tests for cluster database instances, including global cache converts, timeouts, consistent read requests, freelist waits, and so forth. Data from the Oracle Data Gatherer is used to create these Event Management reports.To access the Event Management feature, from the Event menu, select Create Event. The Create Event window opens from which you can select targets, set parameters, and set notification preferences, using the following subpages:

• The General Page is used to select the cluster database and cluster database instance target types.

• The Tests Page is used to select the type of specialized cluster databases tests that you want to schedule.

• The Parameters Page, is used to select the parameter for your test event.• The Schedule Page is used for scheduling the execution of a test event.• The Access Page is used for setting notification options.• The Fixit Jobs Page is used to manage jobs that provide custom responses to events.

The Event Management Tests submenu is shown in the slide.



Parameter Files in Cluster Databases

• You can continue to use client-side initialization parameter files for your Real Application Clusters database

• You can use a single server parameter file for all of your instances– This is the preferred approach– The file must be available to all cluster database nodes

on a shared disk– Allows parameter value changes to persist across

instance shutdowns – You can change parameter values for all instances

with a single ALTER SYSTEM command

Parameter Files in Cluster DatabasesA client-side initialization parameter file for an instance must be located on all of the systems from which the instance is started. In order to provide the parameter values that are unique to each instance, you need a client-side parameter for each instance. You may also want to use a common parameter file, for values that are identical on all instances, and include it in the instance-specific files with the IFILE parameter. You can also use a server parameter file for your Real Application Clusters database, as recommended by Oracle Corporation. This binary file is maintained on a shared disk and contains generic entries for values that are common to all instances and a separate parameter entry for each instance that requires a unique value.If you build your Real Application Clusters database with the DBCA, you have the option to create a server parameter file concurrently with the database. Select the “Create server parameter file (spfile)” box under the “File Locations” tab on the “Initialization Parameters”page and provide the shared disk path name in the “Persistent Parameters Filename” field.You can also create a server parameter file manually if you have built or migrated your Real Application Clusters database without the DBCA or if you did not select the “Create server parameter file (spfile)” option.



Create and Manage Server Parameter File

CREATE SPFILE='/dev/vx/rdsk/oracle/T1_raw_spfile_5m' FROM PFILE='$ORACLE_HOME/dbs/initT1.ora'

CREATE PFILE='%ORACLE_HOME%\database\initspfile.ora' FROM SPFILE='\\.\T1_raw_spfile_5m'

ALTER SYSTEM SET sort_area_retained_size = 131072SCOPE = SPFILE SID = 'T1N1'

ALTER SYSTEM SET sort_area_size = 131072COMMENT = 'Reduce sort space'SCOPE = SPFILE SID = '*'

Create and Manage Server Parameter FileTo create a server parameter file, you first need a text-based, client-side text initialization parameter file and a shared disk. Connect to a SQL*Plus session and execute the CREATE SPFILE command as shown in the first (UNIX) example. If you are using a shared file system, name the default SPFILE location in the command, otherwise name the raw device or a link to it defined with the default file name. You can do this regardless of whether you have a running instance or an open database.When initially created, all parameters in a server parameter file have identical values regardless of which instance uses it to startup. To add instance specific values, you must issue an ALTER SYSTEM command with a SCOPE clause set to MEMORY (or BOTH) and the SID clause set to the required instance name, as shown in the second example. You can also set a database wide value in your server parameter file by setting the SID value to the wild card value ('*') as shown in the third example, which also includes a comment. You can remove parameters from the SPFILE with the ALTER SYSTEM RESET command. To preserve your server parameter in case of disk failures, you should make a text backup of the file with the CREATE PFILE command as shown in the fourth (Windows) example. If you want to convert a server parameter file back to a client-side text file, you should also execute the CREATE PFILE command.



Parameter File Search Order

• The default search order for a parameter file during instance startup is:– On UNIX– $ORACLE_HOME/dbs/spfilesid.ora– $ORACLE_HOME/dbs/spfile.ora– $ORACLE_HOME/dbs/initsid.ora

– On Windows– %ORACLE_HOME%\database\spfilesid.ora– %ORACLE_HOME%\database\spfile.ora– %ORACLE_HOME%\database\initsid.ora

• To simplify instance startups, create a generic pfile.ora file with a single SPFILE entry

Parameter File Search OrderWhen you issue a STARTUP command without identifying a parameter file with the PFILE option, Oracle searches for an appropriate file in the following order:

• An instance-specific server parameter file, spfilesid.ora• A generic server parameter file, spfile.ora• An instance-specific client-side parameter file, initsid.ora

The full directory path to these files is platform dependent. The slide shows the paths for UNIX and Windows systems.Even though server parameter files are in the search list, your Real Application Clusters server parameter file will be on a shared disk and, therefore, not likely to be in the default location with the default name. In order to take advantage of the default behavior to locate your server parameter file, create a text file containing just one line: the SPFILE parameter. The value for this SPFILE parameter is the full name of the shared disk partition where you created the file. By locating and naming this text file as if it were a generic server parameterfile, all instances started on the server will locate and use it during a default startup. You may also be able to use the default behavior by creating a link (to the shared partition where the server parameter file is stored) and giving it the same name and location as the default text generic server parameter file.



Practice Overview: Employing Database Configuration and Management Tools

This practice covers the following topics:• Starting the Global Services Daemon• Configuring the database server management file• Starting and stopping database components with

SRVCTL• Creating and managing a server parameter file

(SPFILE)



Summary

In this lesson, you should have learned how to:• Start the GSD• Store database and instance configuration

information with the SRVCTL utility• Start and stop database components with SRVCTL• Manage cluster databases with Oracle Enterprise

Manager• Create and use PFILEs, IFILEs, and SPFILEs to

manager initialization parameters for Real Application Clusters instances


Scalability and Cache Fusion



Objectives

After completing this lesson, you should be able to do the following:• Define speedup and scaleup• Implement load balancing across instances • Compare Cache Fusion with pre-Oracle9i block

transfer behavior



Levels of Scalability

• Hardware• Operating system• Database • Application

Levels of ScalabilitySuccessful implementation of parallel processing and parallel databases requires optimal scalability on four levels:

• Hardware scalability: Interconnectivity is the key to hardware scalability, which greatly depends on high bandwidth and low latency.

• Operating system scalability: Methods of synchronization in the operating system can determine the scalability of the system. In some cases, potential scalability of the hardware is lost because of the operating system’s inability to handle multiple resource requests simultaneously.

• Database management system scalability: A key factor in parallel architectures is whether the parallelism is effected internally or by external processes. The answer affects the synchronization mechanism.

• Application scalability: Applications must be specifically designed to be scalable. A bottleneck will occur in systems in which every node is updating the same data most of the time.



Scaleup and Speedup

Original system

Hard-ware 100% of task

Time

Cluster system scaleup

up to 200%oftask

up to 300%oftask

Hard-ware Time

Hard-ware Time

Hard-ware Time

50% of task

Cluster system speedup

Hard-ware Time

Hard-ware Time

50% of task

ScaleupScaleup is the capability to provide continued increases in throughput in the presence of limited increases in processing capability while keeping time constant:

Scaleup = (volume parallel) / (volume original) – time for interprocess communicationFor example, if thirty users consume close to 100 per cent of the CPU during their normal processing, adding more users would cause the system to slow down due to contention for limited CPU cycles. However, by adding CPUs, extra users can be supported without degrading performance.

SpeedupSpeedup is the capability to provide continued increases in speed in the presence of limited increases in processing capability, while keeping the task constant:

Speedup = (time original) / (time parallel) – time for interprocess communicationSpeedup results in resource availability for other tasks. For example, if queries would normally take ten minutes to process, and running in parallel reduces the time to five minutes, then additional queries can run without introducing the contention that might occur were they to run concurrently.



Speedup and Scaleup for Different Types of Workloads

Workload

OLTP and Internet

DSS with parallel query

Batch (mixed)

Speedup

No

Yes

Possible

Scaleup

Yes

Yes

Yes

Speedup and Scaleup for Different Types of WorkloadsThe type of workload determines whether scaleup or speedup capabilities can be achieved using parallel processing.OLTP and Internet application environments are characterized by short transactions that cannot be broken down further, and therefore no speedup can be achieved. However, by deploying greater amounts of resources, a larger volume of transactions can be supported without compromising the response. If OLTP applications are not well designed, there may be performance degradation because of the increased overhead required for synchronization.The decision support systems and parallel query options can attain speedup as well as scaleup because they essentially support large tasks without conflicting demands on resources. Oracle's parallel query capability can also be leveraged to decrease overall processing time of long-running queries and to increase the number of such queries that can be run concurrently.In an environment with a mixed workload of DSS, OLTP, and reporting applications, scaleup can be achieved by running different programs on different hardware. Speedup is possible in a batch environment, but may involve rewriting programs to take advantage of the parallel processing capabilities.



Load Balancing with Oracle Net Services

Instances

Database

ListenersClients

Load Balancing with Oracle Net ServicesOracle Net Services provides client load balancing and connection load balancing capabilities across the components of a Real Application Clusters database. Client load balancing randomizes client connections among all listeners servicing a database to avoid overloading an individual listener. Connection load balancing improves connection performance by distributing connections across the components of a Real Application Clusters database based on its current work load. Connection load balancing routes client connections based on a node's processing load and the number of active connections on each instance. In a shared server environment, connections are also made to the least loaded dispatcher available for the selected node and instance. The service naming feature of Oracle Net Services is used to implement connection load balancing. Each service is composed of instances that are, in turn, composed of handlers. Handlers are either dedicated servers or, in a shared server environment, dispatchers.For load balancing:

1. A client program specifies the name of the service to which it wants to connect. 2. The listener finds the least loaded instance in the service.3. The listener finds the least loaded appropriate handler in the instance.4. The listener redirects the client to the optimal handler.



Client Load Balancing

Clients

Listeners

sales.us.acme.com=(DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=on)(ADDRESS=(PROTOCOL=tcp)(HOST=sales1)(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)(HOST=sales2)(PORT=1521)))(CONNECT_DATA= (SERVICE_NAME=sales.us.acme.com)))

Client Load BalancingClient load balancing causes a client to choose one of the listeners in its address list randomly. This distributes the load so that individual listeners do not become overburdened. To enable this feature, define a service in the initialization parameter file and, in tnsnames.ora, set the LOAD_BALANCE parameter to ON, YES, or TRUE and provide aSERVICE_NAME (not INSTANCE_NAME) as shown in the example in the slide.Oracle Net Services employs the Process Monitor (PMON) to register instance information with a listener automatically, so you do not need to configure your listener.ora file with static instance information. To ensure this service registration feature works properly, your initialization parameter file should provide values for the SERVICE_NAMES for the database service name and INSTANCE_NAME for the instance name. For example:

SERVICE_NAMES=sales.us.acme.comINSTANCE_NAME=sales

The SERVICE_NAMES parameter value defaults to the global database name, which is derived from the values from the DB_NAME and DB_DOMAIN parameters in the initialization parameter file. For ease of management, the INSTANCE_NAME should be the same as the SID defined at the operating system level.Note: Client load balancing may not be useful if you implement application partitioning.



Connection Load Balancing

Nodes

sales1.us.acme.com=

(DESCRIPTION=

(ADDRESS_LIST=

(LOAD_BALANCE=on)

(ADDRESS= . . . ))

(ADDRESS= . . . ))

(ADDRESS= . . . )))

(CONNECT_DATA=

(SERVICE_NAME=

sales.us.acme.com)

(SERVER=shared)))

Dispatchers

Connection Load BalancingWith connection load balancing, connections to the handlers on each node are based on the number of active connections; a new connection is assigned to the node with the lightest processing load and the fewest active connections. Further, if you have configured shared servers, the connection will be made to the dispatcher with the fewest current users on the selected node. The example on the slide shows the configuration for connection load balancing across a three-node cluster with shared servers enabled.



Example of Connection Load Balancing

Instance ATotal

Instance BTotal

Load onNode

Dispatcher D1

200 connections

200 connections

60% CPU

Dispatcher D2

100 connections

300 connections

40% CPU

Dispatcher D3

200 connections

Node 1 / LSNR1 Node 2 / LSNR2

Example of Connection Load BalancingIn the example, a Real Application Clusters database has two instances, Instance A on Node 1 and Instance B on Node 2. Listeners LSNR1 and LSNR2 run on Nodes 1 and 2, respectively. Instance A has one dispatcher, identified as D1 for convenience, and Instance B has two dispatchers, labeled D2 and D3.Instance A has the same number of connections (200) as its only dispatcher, D1. The number of connections on Instance B is the sum of the connections (300) on its two dispatchers, D2 (100) and D3 (200). Therefore, Instance B serves more connections than Instance A.Currently, the CPU usage on Node 2 (40 percent), is less than that on Node 1 (60 percent). In this scenario, an incoming client connection request will first be routed randomly to either listener, LSNR1 or LSNR2. The receiving listener will route the connection request to the least-loaded cluster node, Node 2. On Node 2, there is a choice of dispatchers, D2 and D3. The new client's connection will be redirected to D2 because dispatcher D2 is currently supporting fewer connections than dispatcher D3.



Service and Instance Names

(DESCRIPTION=(LOAD_BALANCE=ON)(ADDRESS=(PROTOCOL=tcp)(HOST=host1)(PORT=1521))(ADDRESS=(PROTOCOL=tcp)(HOST=host2)(PORT=1521))(ADDRESS=(PROTOCOL=tcp)(HOST=host3)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=sales.us.acme.com)))

(DESCRIPTION=(ADDRESS= (PROTOCOL=tcp)(HOST=host1)(PORT=1521))(CONNECT_DATA=

(SERVICE_NAME= sales.us.acme.com)(INSTANCE_NAME=S1))

)

Service and Instance NamesThe DESCRIPTION clause in the first example will enable load balancing for connections to the sales.us.acme.com service name. Note that if you omit the LOAD_BALANCEclause, or set LOAD_BALANCE to OFF, NO, or FALSE, the addresses will be tried in the order listed until a successful connection is made. The second example’s DESCRIPTION clause causes a connection to be made specifically to the instance with its INSTANCE_NAME initialization parameter set to the value S1. This option enables connections to a specific instance based on the work being performed while connected. This usage supports functionally partitioned databases.



Adaptive Parallel Query

QuerycoordinatorParallel queryexecution

Query processeshave node affinity for query coordinator…

…but will use other nodes if needed

Node 2 Node 3Node 1

Adaptive Parallel QueryAs well as load balancing provided by Oracle Net Services, you can employ the adaptive parallel query mechanism to execute statements in parallel across the instances of a Real Application Clusters database. This method allows the optimizer to determine if it will spread the work across query processes associated with one instance or with multiple instances. Therefore, depending on the workload, queries, data manipulation language (DML), and data definition language (DDL) statements may execute in parallel on a single node, across multiple nodes, or across all nodes in the cluster database. In some cases, the Oracle parallel optimizer may choose to use only one node to satisfy a request. Generally, the optimizer will try to limit the work to the node where the query coordinator process is executing (node affinity) to reduce cross-instance message traffic. However, if multiple nodes are employed, they all continue to work until the entire operation is completed.



Cache Fusion

• Cache Fusion helps provide transparent scalability in a Real Application Clusters database

• The algorithms enable transportation of block images between instances

• Cache Fusion services track the current location and status of resources

• Directory structures within the SGA of each instance store the resource information

Cache FusionThe Cache Fusion services and structures enable instances to find global resources and database blocks that are currently in use by one or more instances. The services are also responsible for transferring information, including database block images, between instances using the cluster interconnect. In this way, Cache Fusion provides an apparent single cache, consisting of the contents of the individual instance buffer caches, for the database users.Information about the cache resources is maintained in the Global Cache Directory, the contents of which are distributed across the active instances. The instance responsible for the cache directory entry is known as the resource master. The following pages introduce the key components of Cache Fusion and explain their relationship to the database and cluster architecture.



Cache Fusion Model

The Global Resource Directory• Is managed by the Global Cache Service• Records

– Resource modes– Resource roles– Block status (clean or dirty) in the instances

• Distributes the resource masters across the active instances to achieve affinity

• Redistributes resource masters as necessary when instances start up and shut down

Cache Fusion ModelThe Global Resource Directory, which is distributed across the active instances, contains the current status of resources shared by the instances. Its contents are maintained by the Global Cache and Global Enqueue Services using messages containing sufficient information to ensure that the block images currently held in various instance caches can be located. The mode in which block resources are held (NULL, Shared, or Exclusive), as well as the resource role (local or global), are also maintained by the Global Resource Directory. Details of resource modes and roles are discussed on the next pages. The Global Cache Service is responsible for assigning and updating resource modes and roles and locating the most current block image when necessary. The information managed and maintained by the Global Resource Directory allows Real Application Clusters to minimize the time taken to return to normal processing following an instance failure or cluster reconfiguration.



Global Cache Service Resource Modes

Cache resources used by Cache Fusion can be held in one of three modes:• NULL • Shared (S)• Exclusive (X)

Global Cache Service Resource ModesA block resource can be held by an instance in one of three modes:

• NULL (N): The NULL mode is the default status for each instance. It indicates that the instance does not currently hold the resource in any mode.

• Shared (S): The shared resource mode is required for an instance to read the contents of a block, typically to satisfy a query. Multiple instances can hold a shared mode resource on the same block concurrently.

• Exclusive (X): The exclusive resource mode allows an instance to change the contents of a block covered by the resource. When an instance holds a resource in exclusive mode, all other instances must hold it in NULL mode.



Global Cache Service Resource Roles

Cache resources used by Cache Fusion can be held in one of two roles:• Local

– The original mode when a resource is first acquired– Only one instance can hold a dirty copy of the block

• Global– Local resources are converted to global when a block

becomes dirty in more than one instance– Blocks can only be written to disk when directed by

Global Cache Service

Global Cache Service Resource RolesThe global role is used to manage blocks that are dirty in more than one instance concurrently. Furthermore, the dirty block image of a globally managed block can be different from its image in another instance.The global role is initially assigned when a changed block image is served to another instance without an intervening disk write. To ensure that an instance is working with the correct image of a globally managed block, the instance cannot manipulate the block without coordination, through the Global Cache Service, with other instances holding the resource.When a block is globally managed:

• The resource mode can be NULL, S, or X.• It was, or is, a current block in an instance under an X resource mode.• It was dirty when the global role was assigned.• If the mode is X, it can be modified in its current cache.• Its image on disk may or may not be current.• It can be served to another instance when required• A write request, for any reason, must be approved by the Global Cache Service• The Global Cache Service identifies the instance that will perform the disk write.



Cache Fusion Block Transfers:Example Overview

Instance A

1008

Instance D

Resourcemaster

Instance C

Instance B

Cache Fusion Block Transfers: Example OverviewThe following examples show the messages, resource status changes, and block transfers involved in moving blocks between disk and memory and in Cache Fusion transfers of blocks between instances. This slide shows the setup used for these examples.There are four instances, A, B, C, and D, and a shared drive. For simplicity, the examples use just one block which is initially shown on disk with a system change number (SCN) of 1008. The examples show the current SCN for the block on an instance and on disk and also use a single character notation to indicate the block resource status on an instance where

• N (for NULL) indicates the there is no resource status recorded for the block (this status is only shown when an instance is converting to or from another resource status

• S (for shared) indicates a resource that allows the instance, along with other instances, to read the contents of the related block

• X (for exclusive) indicates that the instance is allowed to modify the contents of the associated block.

Note that the SCN and resource status information is actually stored in the Global Resource Directory on the node currently responsible for mastering the block information. The block used throughout these examples is mastered on instance D.



Example 1: Read with No Transfer

Instance A

1008

Instance D

Resourcemaster

Instance C

Instance B

Request toobtain a shared resource on C

N

1

Example 1: Read with No TransferIn this example, instance C requests shared resource on the block in order to read its current contents. The block is currently not available in the cache of any of the instances.Step 1Instance C sends a request for the shared resource to the Global Cache Service. This request is directed to instance D where the resource is mastered.




Instance A

1008

Instance D

Resourcemaster

Instance C

Instance B

The request is granted and the requesting instance is informed

N–>S

1

2

Example 1: Read with No Transfer (continued)Step 2The Global Cache Service grants the resource in shared mode with the local role, records this information in the Global Resource Directory on instance D, and informs instance C of the grant. Instance C converts the NULL status on the resource to shared mode.




Instance A

1008

Instance D

Resourcemaster

Instance C

Instance B

S

Read request

1

3

2

Example 1: Read with No Transfer (continued)Step 3Instance C initiates the I/O with a read request message to the disk for the block.




Instance A

1008

Instance D

Resourcemaster

Instance C

Instance B

S Block imagedelivered

10081

2

3

4

Example 1: Read with No Transfer (continued)Step 4The I/O is completed in step 4 with the delivery of the block to instance C. Instance C now holds the block with SCN 1008 using an S resource.



Example 2: Read to Write Transfer

Instance A

1008

Instance D

Resourcemaster

Instance C

Instance BRequest to obtain an exclusive resource on B

S

1008

N

1

Example 2: Read to Write TransferIn this example, instance B wants the same block as before but, unlike in Example 1, the instance wishes to make changes to the block’s contents rather than just read them. This example starts with instance C holding an S resource on the block which is still at SCN 1008. Note that this is the same status at which the previous example, Example 1, ended. Step 1Instance B sends a request for an exclusive resource on the block to the Global Cache Service on the mastering instance: instance D, as in the previous example (example 1).




Instance A

1008

Instance D

Resourcemaster

Instance C

Instance B

Instruction to transferthe block to B for exclusive access

S

1008

N

2

1

Example 2: Read to Write Transfer (continued)Step 2The Global Cache Service ascertains that the block is being held by instance C and sends a request to that instance. The request asks instance C to transfer the block, for exclusive access, to instance B. In a more complex situation, more than one instance could be holding S resources on the block for which an instance is requesting the exclusive resource. In such cases, the Global Cache Service sends, to all but one of the holding instances, a message to transfer the block to a null location. This effectively tells these instances to close their shared resources on the block and to release the buffers holding the block. Once this is done, the last remaining shared resource holder is equivalent to instance C in this example: it is the only instance holding an S resource on the requested block. At this point, the actions performed by the Global Cache Service and the remaining resource holder are identical to the steps shown in this example, with instance C having the role of the last holder of the requested shared resource.




Instance A

1008

Instance D

Resourcemaster

Instance C

Instance B

Block and resource status (including C’s plan to close its resource)

S–>N

1008

N

1008

2

1

3

Example 2: Read to Write Transfer (continued)Step 3On receipt of the transfer message sent in step 2, instance C does the following:

• Sends the block to B, as requested, along with an indicator that is closing its own resource and supplying an exclusive resource for use by the receiving instance

• Closes its own resource, marking the buffer holding the block image as Consistent Read (CR), identifying it as available for reuse.




Instance A

1008

Instance D

Resourcemaster

Instance C

Instance BResource assumptionand status message

N

1008

N–>X

1009

2

1

3

4

Example 2: Read to Write Transfer (continued)Step 4On receipt of the block message, instance B converts its resource and sends a message to the Global Cache Service. The message includes information about the assumption of the X resource on instance B and the closure of the resource on instance C. Instance B can now modify the block. In this example, the block SCN becomes 1009 following the updates made to the block contents.



Example 3: Write to Write Transfer

Instance A

1008

Instance D

Resourcemaster

Instance C

Instance B

N X

1009

Request to obtain resource inexclusive mode

1

Example 3: Write to Write TransferThis example begins where the previous example (Example 2) ended. Instance B currently holds an exclusive resource for the block being used in these examples. The block is currently in the cache of instance B and it is at SCN 1009. The copy of the block on disk is still at SCN 1008. Instance C does not participate in this example and the copy of the block that could still be in its memory from previous examples is not shown. This is because its buffer is marked CR, that is, the buffer is available for reuse. The resource for this instance was dropped in the previous example (Example 2).This example begins when instance A requests an exclusive resource on the block so that it can modify its contents. Step 1 Instance A sends a request to the Global Cache Service at instance D, where the required resource is mastered, for an exclusive resource on the block.




Instance A

1008

Instance C

Instance B

N X

1009

Instruction to transferexclusive resource to B

1

2

Instance D

Resourcemaster

Example 3: Write to Write Transfer (continued)Step 2The Global Cache Service tells instance B to give up its exclusive resource on the block and to transfer the current image to instance A. This message is sent immediately if the Global Cache Service has completed recording the resource transitions from the previous example (Example 2). If these records are not complete, the request message is queued until the Global Cache Service can process the request.




Instance A

1008

Instance D

Resourcemaster

Instance C

Instance B

N X–>N

10091009

Exclusive-keep copy of buffer

1

2

3

Example 3: Write to Write Transfer (continued)Step 3Instance B completes its work on the block when it receives the message to transfer the block to instance A. This involves:

• Logging any changes to the block and forcing a log flush if this has not already occurred• Converting its resource to N• Sending a copy of the block buffer, at SCN 1009, to instance A along with a notification

that the exclusive resource is available. Because a dirty copy of the block now resides in two different instances, the resource assumes a global role. If there had been no changes to the block’s contents when the message was received by instance B, the instance would simply send the block image to instance A and convert its resource from X to N. This would allow the receiving instance to assume the exclusive mode resource with a local role, just as in the read to write transfer shown in the previous example (Example 2). Note: Global Cache Service resource conversions and Cache Fusion block transfers occur completely outside transaction boundaries. An instance does not have to wait for a pending transaction to complete (commit or rollback) before releasing an exclusive block resource and transferring the block image to another instance.




Instance A

1008

Instance D

Resourcemaster

Instance C

Instance B

N–>X

10091013

N

1

2

3

4

Example 3: Write to Write Transfer (continued)Step 4After instance A has received the block and information about the resource dispositions, instance A sends a resource assumption message, including the resource information from instance B, to the Global Cache Service on the mastering instance, D. This notifies the Global Cache Service that instance A has the resource with an X status and that instance B, the previous holder of the exclusive resource, now holds a block copy at SCN 1009. Instance A is able to obtain the block SCN from the copy sent by instance B because this copy contains all of the changes made by instance B. Once this is done, instance A can modify the block. In the example, the modification converts the block to SCN 1013. Note that because it no longer has an exclusive resource, instance B cannot make any further changes to the block even though the instance maintains a copy in its buffer cache.



Example 4: Write to Read Transfer

Instance A

1008

Instance D

Resourcemaster

Instance C

Instance B

X

10091013

N

NRequest to obtainresource in shared mode 1

Example 4: Write to Read TransferExample 5 continues where the previous example (Example 3) ended. That is, the block image on disk is still at the original SCN, 1008. There is a copy of it in instance B at SCN 1009 and the current copy, at SCN 1013, is in instance A, held with an exclusive resource. In this example, the block is requested by instance C for a current read. Such a request is typically made when the block contents must reflect the most recent version of the data, such as a header block that contains the extent map or an index branch block that points to leaf block value ranges. This request is satisfied with Cache Fusion.Had the request been for a read consistent image of the block, instance A would construct the block image using its own undo data to restore the contents to the SCN required by the query on instance C. The read consistent block image would then be transferred to instance C across the cluster interconnect without changes to the resource mode on instance A. This process is known as Consistent Read Cache Fusion.Step 1The first step is the request by instance C to the Global Cache Service for the necessary shared resource. As before, this request is directed to instance D where the resource is mastered.




Instance A

1008

Instance D

Resourcemaster

Instance C

Instance B

X

10091013

N

NInstruction to transfer shared resource to C

1

2

Example 4: Write to Read Transfer (continued)Step 2The Global Cache Service instructs instance A to transfer a shared resource to satisfy the request from instance C. As before, this message is sent immediately if the Global Cache Service has no resource transactions in progress, otherwise it is queued.




Instance A

1008

Instance D

Resourcemaster

Instance C

Instance B

X–>S

10091013

N

N

1013

Shared-keep copy of buffer3

2

1

Example 4: Write to Read Transfer (continued)Step 3On receipt of the message to transfer the block, instance A completes its work on the block under the current buffer cache pin latch. As in the previous example (Example 3), this may involve logging changes and flushing the log buffer on instance A before sending the block. To satisfy a read request, an exclusive resource is not needed by the receiving instance, so instance A downgrades its resource to shared mode. After this is done, the instance sends a copy of the block to instance C. As well as the current block contents, the message identifies the resources on the block in each instance involved in the process.




Instance A

1008

Instance D

Resourcemaster

Instance C

Instance B

S

10091013

N

N–>S

Resource assumptioninformation

1013

2

3

4

1

Example 4: Write to Read Transfer (continued)Step 4Instance C extracts the SCN from the block it receives from instance A and constructs a resource assumption message for the Global Cache Service. This contains sufficient information to update the Global Resource Directory with new status of the resource on each of the instances, including the SCN of the block on instance A. Instance C sends the completed message to instance D, the instance mastering the resource for the Global Cache Service.



Block Transfers in Real Application Clusters

InstanceA

Node 1

InstanceB

Node 2

Cache Fusion block transfer

Block transfer with forced disk write

Block Transfers in Real Application ClustersWith Cache Fusion, resources in the Global Resource Directory are automatically managed for you. Each block has its own resource so when any change is made to the mode or role of the resource, no other block is affected by the change. Similarly, only one block is involved in a Cache Fusion block transfer and the transfer itself requires minimal resources because the image is simply passed across the cluster interconnect. In some cases, you may want to reduce the amount of storage required to manage the cache resources by assigning locks to multiple blocks. This can reduce overhead, compared to Cache Fusion, if all the blocks covered by a lock do not require a conflicting status on different instances. This is because changing the status (from NULL to S or X to S, and so on) of a multiblock lock automatically updates the status all the blocks under the lock, hence fewer messages are required than to change each block’s status individually.However, improper implementation of multiblock locks can degrade overall database performance. First, the Cache Fusion algorithm for block transfers is disabled. Instead, when a block has to be transferred between instances, the current instance writes it to the data file on disk and the other instance has to read it from disk. Second, down-converting a lock from an X status requires all dirty blocks covered by the lock to be written to disk. Consequently, you need to monitor, and be prepared to tune, the lock assignments.



Configure Multiblock Locks

Set the parameter GC_FILES_TO_LOCKS with the following components enclosed in double quotes:• One or more file ID lists with a colon separator (:)• A lock count for each list• An optional grouping factor for a list designated

with a exclamation point (!)• The optional keyword EACH for a list

GC_FILES_TO_LOCKS =\

"2=200:12,14-16=1000!50:4,5-8,13,18-20=5000EACH"

Configure Multiblock LocksUse the initialization parameter GC_FILES_TO_LOCKS to assign a set of locks to one or more data files. These locks are used in place of cache resources and are held in NULL, shared, or exclusive mode by an instance. The syntax of the GC_FILES_TO_LOCKSparameter value is

"{file_list=lock_count[!grouping_factor][EACH][:]}..."where:file_list is a single file ID number, a comma separated list of file IDs, a range of file IDs, or combinations of these elements. Query the view DBA_DATA_FILES to identify file ID numbers. The example has three file lists, the second contains file IDs 12, 14, 15, and 16. lock_count is the number of locks to be assigned across the files listed. Each lock covers zero, one, or more than one blocks in each file in file_list, depending on the number of blocks in each file. In the example, the second set of files share 1000 locks.!grouping_factor causes each lock to cover the defined number of contiguous blocks. The default value is one, meaning that adjacent blocks are covered by different locks. In the example, each lock in the second file list covers one or more sets of 50 adjacent blocks.EACH assigns a separate group of lock_count locks to each file in the list. The assignment for the third list of files in the example defines 45,000 locks, 5000 for each of the nine files.



Notes on Using Multiblock Locks

• Use only for query intensive or already-partitioned databases

• Assign the same value to GC_FILES_TO_LOCKS for every instance in your cluster database

• Leave no white space in the parameter value between the double quotes: separate the entries for each set of file IDs with a colon

• Configure Cache Fusion, instead of multiblock locks, for selected files by– Not including their IDs in GC_FILES_TO_LOCKS– Setting their lock_count to zero and not assigning a

grouping factor

Notes on Using Multiblock LocksYou should consider multiblock locks only if your database is query intensive or is already partitioned for access by different instances. This is because you need to plan your database structure and contents (which segments are assigned to each tablespace or file) and continue to monitor block usage to ensure there are minimal disk transfers between instances.Every instance must use the same locking strategy for each data file so, if you use GC_FILES_TO_LOCKS, you must assign the same value for each instance. The value clause of the parameter must be enclosed in double quotes and contain no white space within the quoted string. If the value spans multiple lines in your initialization file, you can use a continuation character adjacent to the last character of the continued line, as shown on the previous slide. If you need to include multiple sets of files in order to assign different lock granularity (numbers of blocks covered by a lock), concatenate the substrings for each file list in the same value clause using a colon separator, again without leaving any white space.The default value of GC_FILES_TO_LOCKS is equivalent to "0=0" which causes all files to be managed with Cache Fusion. Only files identified in one of the file lists are managed with multiblock locks. You can include IDs for files that are not part of the database in GC_FILES_TO_LOCKS so they will use multiblock locks if they are added but, until that time, your cluster database will have to master these unused locks.



Practice Overview: Working Across Multiple Instances

This practice covers the following topics:• Examining intra- and inter-instance parallelism• Client load balancing• Comparing overhead without Cache Fusion using

various locking strategies• Comparing overhead without and with Cache

Fusion



Summary

In this lesson, you should have learned how to:• Define scaleup and speedup• Configure load balancing• Assign and use instance and service names• Configure adaptive parallel query• Identify basic Cache Fusion operations• Employ multiblock locks and Cache Fusion

appropriately


High Availability Considerations



Objectives

After completing this lesson, you should be able to do the following:• Set parameters to minimize instance recovery

overhead• Use Oracle9i Data Guard with a Real Application

Clusters database• Recognize components of Oracle Real Application

Clusters Guard architecture



High Availability Features

Real Application Clusters is Oracle’s premier high-availability solution with the following abilities:• Detection and circumvention of network problems

without intervention• Reconfiguration and recovery from failures with

minimal disruption • Distribution of work from a failed node to other

nodes in the cluster

High Availability FeaturesHigh availability is an important consideration in databases where the cost of down time is very high due to loss of business. Such databases include Web servers, particularly for companies that provide services or products over the Web for profit, and corporate databases that manage key daily operations. In Real Application Clusters, high availability is provided by the ability of surviving instances to continue processing when one or more instances fails.To provide the highest availability, the amount of time between an instance failure and resumption of work on an alternate instance must be minimized. Real Application Clusters provide the following capabilities to help achieve this goal:

• Cluster-aware code is included in the database software, allowing the instances to detect cluster problems without having to coordinate with the system’s cluster management software

• When a problem is detected, instances communicate among themselves to determine their status and decide which instances are going to remain active members of the cluster

• After a new set of active instances is determined, these instances reconfigure and redistribute database resources to minimize impact on existing work and allow work from the failed resource to resume.



Cache Resource Remastering

• Cache resources need not continue to be mastered on the node to which they were initially assigned

• An activity called dynamic remastering can move them to a different node

• The Global Cache and Global Enqueue Services use dynamic remastering – To move resources to instances where they are being

used frequently– To redistribute resources after a new instance joins

the active set

Cache Resource RemasteringTo alleviate bottlenecks on one instance, the Global Cache Service and Global Enqueue Service resources are evenly distributed among the available instances. However, this pattern can be changed for cache resources. In some cases, cache resources are dynamically remastered by reassigning them to a new mastering instance.The primary cause of dynamic remastering is when cache resources associated with a specific data file are requested by just one instance over a period of time. In such cases, the resources for this file are remastered on the requesting node to avoid further inter-instance communication to provide the resource.The other type of activity that causes resource remastering is a cluster reorganization. This occurs when an active instance leaves the Real Application Clusters database or when a new instance joins. When a new instance joins the cluster, the resources are gradually redistributed across the active instances, including the new one, based on the workload. This avoids having to remaster all of the resources during a node transition and therefore reduces the time taken for such a transition. This can be particularly important after an instance failure when application users are wanting to continue work as quickly as possible.



Automatic Resource Remastering

• Tablespace access by instance is tracked automatically

• Tablespaces used by only one instance are flagged• The Global Cache Service resources associated

with the data files belonging to such tablespaces are remastered on the instance using them

• If the same blocks are consistently remastered to the same instance after startup, consider the use of large multiblock locks

Automatic Resource RemasteringAn internal mechanism tracks which instances access each tablespace. If a single instance is identified as the sole user of a tablespace, the block resource masters for that tablespace’s files are lazily moved to that instance. This reduces the overhead for that instance to open these resources because no messages need be sent to the Global Cache Service on another node.It is relatively easy for Oracle Real Application Clusters to determine if a tablespace is being accessed by only one instance, and moving resource masters from various nodes in a cluster to a single instance requires little overhead. As a result, automatic remastering is robust and transparent.You can identify remastering activity by querying the local dynmic performance view, V$GCSPFMASTER_INFO, or the global version, GV$GCSPFMASTER_INFO. Tablespaces consistently selected for automatic remastering are good candidates for less granular resources, defined by GC_FILES_TO_LOCKS, to reduce the resource remastering overhead of the related resources. However, unless you can reliably identify such tablespaces, you are encouraged to allow dynamic remastering perform your resource tuning rather than manually tuning resources with GC_FILES_TO_LOCKS.



LMON and Cluster Reorganization

LMON• Communicates with the Cluster Manager to provide

a current cluster membership to the instances• Validates the health of Real Application Clusters

instance configuration with a disk-based heartbeat and a voting procedure– Each member's LMON process gives its impression of

the other members’ availability– The current status is maintained on disk

• Initiates recovery actions if a node or instance fails to send or acknowledge a heartbeat message

LMON and Cluster ReorganizationReal Application Clusters relies on both the Cluster Manager software and its own mechanisms for failure detection. The Cluster Manager maintains heartbeat functions that periodically assess the status of the cluster nodes. On most systems, you can configure the heartbeat timer with a system-specific parameter. Note that low timer interval is more likely to report false cluster failures due to transient problems. LMON also provides cluster status checking by continually sending messages from the node it runs on and regularly writing to a shared disk. The absence of these functions from any node is evidence to the surviving nodes that the node is no longer a member of the cluster.When a node fails, Oracle must perform a cluster reorganization. LMON initiates the recovery actions that include remastering of Global Cache Service and Global Enqueue Service resources and instance recovery. At this stage, the Real Application Clusters environment is in a state of system pause, and most client transactions suspend until Oracle completes recovery processing. The cluster reorganization should be a very fast operation and cause only minimal impact on the clients.



Example

Instance A

Node 1

Global Resources

ID…20232629…

Grantedto

Instance…

A,B,CA,BC

B,C…

Instance B

Node 2

Global Resources

ID…21242730…

Grantedto

Instance…B

A,CA,B,CB,C…

Instance C

Node 3

Global Resources

ID…22252831…

Grantedto

Instance…

A,CC

A,B,CB,C…

ExampleThe Real Application Clusters environment for this example consists of three instances, A, B, and C, running on nodes 1, 2, and 3 as shown. For each instance, a subset of the Global Cache Resources being mastered is shown, along with the names of the instances to which the resources are currently assigned.For clarity, the diagram does not contain a lot of detail. The Global Resource Directory, for example, maintains much more information than the instance name in its list of granted resources.



Example

Instance A

Node 1

Global Resources

ID…202123262729…

Grantedto

Instance…

A,C

AC

A,CC…

Instance C

Node 3

Global Resources

ID…222425283031…

Grantedto

Instance…

A,CA,CC

A,CCC…

Instance B

Node 2

Global Resources

…21242730…

…B

A,CA,B,CB,C…

Grantedto

InstanceID

Example (continued)In the next step of this example, node 2 crashes causing instance B on that node to fail. This requires that the resources being managed on instance B be remastered on one of the surviving instances. The diagram shows the remastering of resources 21 and 27 to instance A and resources 24 and 30 to instance C. As the resources are remastered, they are cleared of any reference to the failed instance. Therefore, as the diagram shows, the resource information on node 2 indicates the status prior to the remastering while the information on nodes 1 and 3 indicates the status after the resources have been remastered and their grant lists have been updated.



Instance Transition and Recovery

• If an instance crashes – Resource information mastered by that instance is lost– During reconfiguration, lost resource information has

to be reconstructed– Recovery of the failed instance is required

• If an instance shuts down normally, no recovery operations are required

Instance Transition and RecoveryShould an instance leaves the cluster in an abnormal manner, caused, for example, by a server or interconnect failure or the execution of a SHUTDOWN ABORT command, the portion of the Global Resource Directory being mastered on that instance is lost. In addition to remastering the resources, the surviving instances must also rebuild the resource status information that was lost when the instance failed. Finally, one of the surviving instances must perform instance recovery from the redo logs of the failed instance in order to restore transaction consistency to the database. Until these operations are complete, the recovery domain, that is, the set of information needed to perform recovery should an instance fail, remains inconsistent and a further shutdown will require additional processing to restore the database.When you perform a normal shutdown of a Real Application Clusters instance, that is, you execute a SHUTDOWN command with an explicit or implicit NORMAL option or with an IMMEDIATE or TRANSACTIONAL option, no cache recovery operations are required.



Global Resource Directory Reconfiguration and Cache Recovery

Recovery time

GlobalCacheService

SMON onRecoveringInstance

Rebuild enqueueresources

Recover cacheresources

Build re-covery set

Perform blockrecovery

Global Resource Directory Reconfiguration and Cache RecoveryThe Global Resource Directory contains two groups of resources: enqueue resources, managed by the Global Enqueue Service, and buffer cache resources, managed by the Global Cache Service. During cluster reorganization, the Global Enqueue Service first tries to rebuild the enqueue resources. During this operation, no new enqueue resources are granted and so operations on enqueue resources without a master are temporarily halted. After the Global Enqueue Service completes its recovery of the enqueue resources, the SMON process on the node assigned to perform recovery for the failed instance can obtain the instance recovery enqueue and begin building the recovery set (the set of blocks needing recovery). Meanwhile, the Global Cache Service recovers the buffer cache resources.If SMON finishes building the recovery set before the Global Cache Service finishes recovering cache resources, SMON has to wait. Otherwise, SMON begins claiming the cache resources it needs to perform block recovery during which time other cache resource operations are temporarily frozen. Once SMON finishes claiming the necessary resources, it notifies the Global Cache Service. The recovery domain is then validated, provided there is no additional reconfiguration. All normal operations are then resumed and reconfiguration completes.



High Availability Design Considerations

• Create and test policies for change management• Configure redundant hardware• Provide primary/secondary or spare nodes• Set appropriate instance initialization parameter

values• Employ features and options such as

– Oracle9i Data Guard– Real Application Clusters Guard– Transparent application failover

High Availability Design ConsiderationsAvoid application and system software errors by providing an adequate development and test environment and ensuring that all changes are planned, documented, and controlled.Hardware redundancy helps avoid single points of failure. While Real Application Clusters software allows you to take advantage of clusters with multiple servers, you should also consider secondary cluster interconnects and mirrored disk storage.To ensure the same level of service following instance failure, you should provide an identical instance for applications to access. This means employing a simple primary/secondary instance configuration. There are a number of instance configuration parameters that can help reduce the time it takes for a cluster reconfiguration and cache recovery to complete. In some cases, you need to decide whether you want settings that help provide high availability or some other characteristic that is more important to your applications.You may also want to consider configuring features and options provided specifically to help maintain highly-available Oracle databases: Oracle9i Data Guard, Real Application Clusters Guard, and transparent application failover (TAF). TAF is covered in the following lesson.



Change Management

• Plan changes to minimize down time and service disruption– May mean overnight or weekend work– Avoid critical periods in application cycles, such as

month- or year-end processing• Consider staged changes

– One function at a time– One node at a time (if possible)

• Include time and resources to back out changes if necessary

Change Management Most change management recommendations to minimize down time and disruption to your applications are no different on a cluster database system than on any other system. They include planning the changes to coincide with periods of light user activity and including time to back out of the changes within the designated time frame. However, when changes involve a cluster and a Real Application Clusters database, you may have the opportunity to stage your changes. For example, if you have partitioned your applications by instance, you may be able to upgrade the part of the application that runs on one instance without upgrading the other pieces that run on other instances. While Real Application Clusters does not support rolling upgrades to the database server (that is, you cannot upgrade your Oracle binaries on one node while continuing to run against an earlier version on another node), you may be able to upgrade your operating system or cluster software this way. Also, if you can temporarily survive with fewer instances, you can shut down some of your instances and upgrade on those nodes (either an Oracle or an operating system release) while continuing to run the earlier release on your other nodes. You can then shut down those instances while you upgrade the remaining nodes but continue to run your applications on the upgraded nodes.



Configure Redundant Hardware

• Node redundancy is an inherent feature of clusters• Interconnect redundancy is a recommended option• Disk mirroring will prevent a disk from becoming a

single point of failure– Recommended for all database files– Recommended for software when using a cluster file

for the Oracle9i binaries

Configure Redundant HardwareReal Application Clusters environments are fully redundant because all nodes access all the disks in the cluster. As long as the cluster has one surviving node, all database clients can process all transactions.Interconnect redundancy is often overlooked in clustered systems. This is because interconnects rarely fail and a redundant cluster interconnect can be cost prohibitive with insufficient business justification. However, a redundant cluster interconnect is an important aspect of a fully redundant cluster and can help protect against not just hardware failures but also against human error.Real Application Clusters operate on a single image of the data, that is, all nodes in the cluster access the same set of data files. In this regard, Real Application Clusters are no different from single instance Oracle. Therefore, you are encouraged to use hardware-based mirroring to maintain redundant media. Operating system software, including the cluster management software, is installed on all nodes of the cluster and hence is implicitly redundant. The same is true of the Real Application Clusters software, unless you are using a cluster file where each instance shares the same Oracle executables. In the latter case, you should ensure you are mirroring the disks where your software is stored.



Primary and Secondary Instances

ClientsNodes

Listeners

. . .*.active_instance_count = 1*.cluster_database_instances = 2sales1.instance_name = sales1sales2.instance_name = sales2. . .

Primary instance: first one tostart

Secondary instance: second one to start; can become primary if other instance fails

Primary and Secondary Instances If you can support the additional hardware, you can provide very efficient options to support highly available Real Application Clusters. You do this by maintaining one or more spare nodes that are used only when one of your normal instances fails for any reason.For a simple two-node cluster, you can employ a special configuration that designates one instance as the primary instance and the other as the secondary instance. In this configuration, database connection requests made through a listener will only be routed to the primary instance. The entire Global Resource Directory will be mastered on this node, providing very efficient processing because there will be no inter-instance communication. However, the secondary instance can be accessed through server-side connections, typically made by the database administrator to perform maintenance and monitoring operations, including database backup and media recovery. If the primary instance fails, the secondary instance automatically assumes the primary role and any connections made through the listener will be routed to this instance. If you restart the failed instance, it will become the secondary instance. To configure your database this way, provide a value of 1 in the ACTIVE_INSTANCE_COUNT initialization parameter of both of your instances.



Spare Nodes

ClientsActivenodesListeners

Sparenodes

Spare NodesAlthough the Transport Network Services (TNS) listeners cannot fence off an instance in a multi-instance Real Application Clusters database like they can for a two-instance database, you can still provide a spare node for use in a fail over situation. You can configure your TNS tnsnames.ora file to provide an alternate instance name for connections that are disconnected from their normal instance. For large clusters, you may wish to save more than one spare node in case you have multiple instances failing at the same time.Whenever you consider a spare node option, the spare nodes must be identical to the nodes they are to replace when there is a problem. If they are smaller or less efficient nodes, the application clients will experience degraded service while working on the spare nodes. Similarly, you should not depend on the spare nodes to perform essential processing on a regular basis; they will become overloaded when also supporting the workload from a failed instance.



. . .

active_instance_count = 1

dml_locks = 0 #useful pre-Oracle9i

fast_start_mttr_target = 30

gc_files_to_locks = "0=0"

recovery_parallelism = 8

service_names = sales.us.acme.com

. . .

Parameters for High Availability

Parameters for High AvailabilityA number of initialization parameters can impact how efficiently and quickly instance recovery will complete. Fast recovery in a Real Application Clusters database is a key factor in how quickly resources become available following the failure of one or more instances. In some cases, you need to balance your need for recovery speed against normal processing performance because some of these parameters affect both. The parameters of particular interest include:•active_instance_count: You use this parameter to enable or disable a primary

and secondary instance configuration as discussed earlier in this lesson.•dml_locks: By setting the value of this parameter to zero, you reduce the number of

resources that need to be mastered and, consequently, remastered during a cluster reorganization. This, in turn, allows instance recovery to begin, and hence to finish, a little faster. However, a zero setting prevents DDL commands such as DROP TABLE, CREATE INDEX, and LOCK TABLE. You can also control DDL at the individual table level with ALTER TABLE…ENABLE | DISABLE TABLE LOCK commands.

Note: Earlier releases could benefit from having DML locks disabled, but you should remove the restrictions imposed by setting dml_locks=0 after you migrate to an Oracle9idatabase.




. . .


dml_locks = 0





. . .

Parameters for High Availability (continued)•fast_start_mttr_target: This parameter sets a target number of seconds within

which instance recovery operations should complete. For a Real Applications Cluster instance, a low value will allow a surviving instance to recover this instance more quickly following a failure. However, lower values incur additional processing overhead so you may want to avoid setting the value too aggressively. Note that setting either fast_start_io_target (a deprecated parameter) or log_checkpoint_interval overrides a setting for fast_start_mttr_target. If you have partitioned your application so that various instances perform different functions, you can set distinct values for fast_start_mttr_target that are appropriate for the workload and any fail over requirements of each instance.

•gc_files_to_locks: As discussed in an earlier lesson, you should only set this parameter after you have determined it could be useful for the overall performance of your database. However, the number of cache resources that have to be remastered as part of instance reconfiguration does have an impact on recovery time and this resource count is either based on Cache Fusion (the default strategy for a data file) or the number of multiblock locks set by this parameter.




. . .


dml_locks = 0





. . .

Parameters for High Availability (continued)•recovery_parallelism: With this parameter, you enable the recovery process to

enlist the help of additional processes to complete instance recovery in parallel. The maximum available degree of parallelism is constrained by the value set in the parallel_max_servers parameter. Setting the recovery_parallelism parameter to a value other than zero or one,which causes serial recovery, will allow an instance to complete parallel recovery on behalf of a failed instance. This should minimize the time before normal processing can resume.

•service_names: The value of this parameter is included in various TNS configuration files to enable a number of advanced features, including load balancing and automatic failover from a failed to an active instance (discussed in a later lesson). Generally you should not need to provide a value for this parameter, which defaults to values of the parameters db_name and db_domain with a period as a concatenation character, to enable automatic failover functionality. However, if you expect to use those features, it does help to have the service name documented in the instance parameter file.



Oracle9i Data Guard

Clients

Primarydatabase

Listeners

Standbydatabase

Oracle9i Data GuardAlthough Real Application Clusters provide you with multiple instances to avoid loss of database access, a truly available system need to address situations in which you may need to process your applications at another physical location. This capability is provided by the Oracle9i Data Guard feature. Oracle9i Data Guard allows you to maintain a standby copy of your database at a remote site. This database is updated with log information generated from your primary database, propagated across an interconnect between the two sites. Ideally, a standby site should have identical hardware to the primary site, thus providing support for the same processing load. However, this can be an expensive option and sometimes a smaller system is selected for the standby to support only the most critical processing. Although this limits your use of a standby site for some administrative options, it may not be within the purview of a database administrator to make this decision.It is possible to use a single instance database as a standby for a multi-instance Real Application Clusters database as shown in the graphic. The standby database site does not require the Real Application Clusters software to be installed; the non-cluster Oracle9isoftware is able to read and manage input from the multiple threads of redo generated by your Real Application Clusters database. Of course, a Real Application Clusters database can be used at the standby site.



Standby Database Real Application Clusters Support

Primary instance A

Primary instance B

Standby receiving instance C

Standby recovery instance D

Archived logs

Archived logs

Onlinelogs

Standbylogs

Archived logs

LGWR RFS ARCn

LGWRRFS RFS ARCn RFS

Standby Database Real Application Clusters SupportThe basic configuration for a Real Application Clusters primary and standby database is illustrated in the graphic. The remote file server (RFS) processes are Data Guard processes that receive log records from the sending process, either log writer (LGWR) or archiver (ARCn). For cross-instance archival processing, a set of standby redo log files (an intermediate set of files that accumulate log records for subsequent processing into the standby system) is required. Standby redo logs can only be written by LGWR processes, as shown.When you use a Real Application Clusters database as your standby database, any instance can receive archived logs from the primary database; this is the receiving instance. However, the archived logs must ultimately reside on disk devices accessible by the node on which the managed recovery operation is performed; this is the recovery instance. Transferring the standby database archived logs from the receiving instance to the recovery instance is achieved using the cross-instance archival operation, performed on the standby database. This operation allows you to separate the log transport services processing on the primary database from the log apply services processing on the standby database, resulting in improved overall performance on both databases.



Log Transport Services: Standby Database Set Up

• Create the standby redo logs• Define the archived log destination to archive

locally on the recovery instance where the MRP executes

• Define, on the receiving instance, the archived log destination to archive to the node where the MRP executes

• Start the ARCn process on all standby database instances

• Start the MRP on the recovery instance

Log Transport Services: Standby Database Set UpTo set up a standby database in a Real Application Cluster environment you need to perform the following steps on the standby database to prepare the log transport services:

• Create the standby redo logs. The standby redo logs must reside on disk devices shared by all instances, such as raw devices.

• On the recovery instance where the managed recovery process (MRP) is to operate, define the archived log destination to archive locally, because cross-instance archival is not necessary. This is accomplished using the LOCATION attribute of the LOG_ARCHIVE_DEST_1 initialization parameter. Unlike a primary database, a standby database is not required to have a local archived log destination.

• On the receiving instance, define the archived log destination to archive to the node where the MRP is to operate. This is accomplished using the SERVICE attribute of the LOG_ARCHIVE_DEST_1 initialization parameter.

• Start the ARCn process on all standby database instances.• Start the MRP on the recovery instance.



Log Transport Services: Primary Database Setup

• Designate the LGWR process to perform the archival operation on all instances

• Designate the standby database as the receiving node

Log Transport Services: Primary Database SetupPerform the following steps to set up log transport services on the primary database:

• On all instances, designate the LGWR process to perform the archival operation.• Designate the standby database as the receiving node. This is accomplished using the SERVICE attribute of the LOG_ARCHIVE_DEST_n initialization parameter.

Note: Ideally, each primary database instance should archive to a corresponding standby database instance. However, this is not required.



Example

ALTER SYSTEM SET LOG_ARCHIVE_DEST_2 = 'SERVICE=prmy1 MANDATORY REOPEN=300';ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_2 = enable;

ALTER SYSTEM SET LOG_ARCHIVE_DEST_1 = 'LOCATION=archivelog MANDATORY REOPEN=120';ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_1 = enable;

ExampleDestination 1 is the repository containing the local archived redo logs required for instance recovery. This is a mandatory destination. Because the expected cause of failure is lack of adequate disk space, the retry interval is 2 minutes. This should be adequate to allow the DBA to purge unnecessary archived redo logs. Notification of destination failure is accomplished by manually searching the primary database alert log. Destination 2 is the primary database on the instance where RMAN is used to back up the archived redo logs from local disk storage to tape. This is a mandatory destination, with a reconnect threshold of 5 minutes. This is the time needed to fix any network-related failures. Notification of destination failure is accomplished by manually searching the primary or standby database alert log. If both your primary and standby databases are in a Real Application Clusters configuration, and the standby database is in managed recovery mode, then a single instance of the standby database applies all sets of logs transmitted by the primary instances. In this case, the standby instances that are not applying redo cannot be in read-only mode while managed recovery is in progress; in most cases, the non-recovery instances should be shut down, although they can also be mounted.



Switchover Considerations

SQL> SELECT instance_name, host_name 2 FROM gv$instance 3 WHERE inst_id <> 4 (SELECT inst_id FROM v$instance);

INSTANCE_NAME HOST_NAME ------------- ---------INST2 standby2

SQL> CONNECT /@standby2 AS SYSDBAConnected SQL> SHUTDOWN. . .SQL> EXIT

Switchover ConsiderationsWhen your database is using Real Application Clusters, active instances prevent a switchover from being performed. You must shut down all other instances prior to performing the switchover operation. You can identify which other instances are active, besides the one where you are connected to perform the switchover, by querying the V$INSTANCE and GV$INSTANCE views, as shown in the first example.In this first example, instance INST2 is still active on a different standby node than the one from which the switchover is being initiated. You can shut down this instance for your current SQL*Plus session as shown in the second example.



Switchover Considerations

SQL> ALTER DATABASE 2 COMMIT TO SWITCHOVER TO STANDBY;

ALTER DATABASE COMMIT TO SWITCHOVER TO STANDBY;

ORA-01105: mount is incompatible with mounts by other instances

Switchover Considerations (continued)If you failed to shut down all but your switchover instance, your attempt to perform the switchover would fail with the following error message as shown in the example:

ORA-01105: mount is incompatible with mounts by other instances

Sometimes, it may be faster and more convenient to recover the primary database rather than perform a switchover. When the primary database is operating in no-data-loss mode and a system failure occurs, you can restart the primary database without incurring data loss on the corresponding standby databases. The following failure scenarios are automatically recovered by log transport services:

• Instance recovery occurs in an open primary database when one instance discovers that another instance has failed.

• Failure recovery occurs when all instances of a Real Application Clusters primary database fail. In failure recovery, the first instance to open the database after a failure generally executes recovery operations.

In both of these scenarios, the standby databases participating in the no-data-loss environment will be automatically resynchronized with the recovered primary database.



Real Application Clusters Guard Architecture

Monitors

VendorCMS

IP addressesListeners

Oracle instance

DGM

Application ormiddleware

Monitors

VendorCMS


Oracle instance

DGM

IP (A) IP (B)

NodeB

(Secondary)

NodeA

(Primary)

Pack Pack

Disk group manager, onlyneeded on some platforms

Real Application Clusters Guard ArchitectureThe Real Application Clusters Guard architecture combines vendor cluster high-availability features with components from Oracle9i Real Application Clusters. The hardware vendor provides cluster architecture that allows each primary node to have a secondary node to which the primary can failover. Although applications can run on secondary nodes, they may not have full functionality. For example, a Real Application Clusters instance on a secondary node may be able to maintain a copy of the library cache as used by the instance on the primary node (a warm library cache), it may not be able to read or write the database files.In addition to the normal Oracle processes, each instance on a node is associated with a pack. A pack is a configuration entity that controls the behavior of Real Application Clusters Guide nodes. It ensures that the database is highly available by managing resources such as database instances, listeners, and IP addresses. The pack uses a set of monitors to ensure that the required components are running correctly and can initiate restarts of failed components when necessary. Pack software can also coordinate a failover to the secondary node and enable client access to the instance on that node if a configurable number of restart attempts fail.



Real Application Clusters Guard Failover

VendorCMS


Oracle instance

DGMMonitors

VendorCMS


Oracle instance

DGM

Heartbeat monitorListener monitorInstance monitor

NodeA

NodeB

(New primary)

Pack Pack

Application ormiddleware

IP (A) IP (B)

Failover

Monitors

Real Application Clusters Guard FailoverThe graphic shows the Oracle Real Application Clusters Guard architecture for a two-node cluster. Node A is the primary node, and Node B is the secondary node. Each node contains:

• A Cluster Manager that executes the run and halt scripts automatically upon failover or as a consequence of a user command

• A pack that controls the resources available on its node The resources on each node include:

• Oracle instance, but only the one on the primary instance is available to clients • Listeners, including a private one used by the pack and one or more public ones used by

clients to access the instance• Public IP addresses, associated with the public listeners• Monitors that manage the instance and listeners and check the status of the other

instances with a heartbeat mechanism• Disk group manager, used only on some platforms, to fence the disks from the secondary

nodeDuring failover, the primary instance role moves from Node A to Node B, making Node B the new primary node. The public IP addresses also migrate to Node B so that clients are connected to the listeners associated with the new primary instance.



Practice Overview: Tuning Instance Recovery

This practice covers the following topics:• Modifying parameters related to instance recovery• Testing instance recovery times• Configuring primary/secondary instances



Summary

In this lesson, you should have learned how to:• Design a Real Application Clusters database for

high availability• Initialize instances in a primary/secondary

configuration• Set initialization parameters to reduce cluster

reorganization times• Employ Oracle9i Data Guard with Real Application

Clusters• Understand the functions of Oracle Real Application

Clusters Guard architectural components


Transparent Application Failover



Objectives

After completing this lesson, you should be able to do the following:• Identify transparent network substrate (TNS)

components involved in Transparent Application Failover (TAF)

• Compare TAF options– Type– Method– Backup– Connection attempts

• Configure TAF



Listeners

Multiple listeners enable• Client load balancing• Connect-time failover• Transparent Application

Failover (TAF)

Clients Cluster nodes

Listeners

ListenersThe listener provides the link between requests to connect to the database made across the Transparent Network Substrate (TNS) of Oracle Net Services and the database itself. A listener is configured listener.ora file. Listeners are started, stopped, and managed with the LSNRCTL command interface, either directly or through tools such as the SRVCTL utility.In Real Application Clusters, multiple listeners on multiple nodes can be configured to handle client connection requests for the same database service. A multiple-listener configuration also enables you to leverage the following failover and load balancing features:

• Client load balancing• Connect-time failover• Transparent application failover

These features can be implemented either singly or in combination with each other.



TNS Connect Descriptors

rac1=(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=aaacme1)(PORT=1630))

(CONNECT_DATA=(SERVICE_NAME=rac.us.aaacme.com)))

$ sqlplus qualassur1@rac1SQL*Plus: – Production(c) Copyright 2000 Oracle Corporation. All rights reserved.Enter password:

TNS Connect DescriptorsA connect descriptor identifies a set of parameters that allow clients to connect to a database through one or more listeners. The parameters identify various characteristics of the connections, including the physical node and port number where the listener is active, information about the database being sought, and the characteristics of the connections to be made.Although you may include the full TNS connect descriptor information as part of a database login string, it is much easier to use a connect descriptor alias. To simplify the management of connect descriptor aliases, use a tnsnames.ora file which associates an alias with the full connect description. A tnsnames.ora file can contain multiple aliases and must reside on the node from which the database connection is being initiated.A simple example of a connect descriptor, with its associated alias, is shown in the sample entry from a tnsnames.ora file. To connect to a database, the connection includes the normal information plus the name of the alias as shown in the second example.Note: You should use the Network Configuration Assistant to create the files for your Oracle9i Real Application Clusters Net Services. If necessary, you can modify them manually. For brevity, examples in this lesson have been edited to remove comments and parameters that are not relevant to the current topic.



Network Naming Methods

• Names resolution service• Local naming using a tnsnames.ora file• Directory naming with a centralized LDAP-compliant

directory server• Java Database Connectivity (JDBC) Driver

– OCI driver for client side and application Web server use

– Thin driver for client side use without an Oracle installation

Network Naming MethodsSelect the appropriate naming method for mapping names to connect descriptors depending on the size of your organization. For a small organization with only a few databases, use host naming to store names in an existing names resolution service, or local naming to store names in tnsnames.ora file on the clients. For a large organization with several databases, use directory naming to store names in a centralized LDAP-compliant directory server.Java client applications access an Oracle database through a Java Database Connectivity (JDBC) Driver, a standard Java interface for connecting from Java to a relational database. Oracle Corporation offers the following drivers:

• OCI driver for client side and application Web server use with an Oracle client installation

• Thin driver for client side use without an Oracle installation, particularly with applets.



Oracle Call Interface

TAF provides failover for:• OCI programs• Java JDBC thick drivers (OCI drivers)• ODBC connections• SQL*Plus• Select statements

Oracle Call InterfaceTAF requires that the connecting application is runs the Oracle call interface (OCI). This includes the following programs:

• OCI programs• Java JDBC thick drivers (OCI drivers)• ODBC connections• SQL*Plus connections

Additionally, TAF provides failover with a continuation of the current operation when the client is executing a SELECT statement.TAF does not provide failover for the following:

• PL/SQL procedure states• Alter session statements• Server side program variables• Ongoing (uncommitted) transactions.



Host-Based Failover

• Detect failure by monitoring the heartbeat• Reorganize cluster membership in the Cluster

Manager• Transfer disk ownership from the primary node to a

secondary node• Restart application and database binaries• Perform application and database recovery• Reestablish client connections to the failover node

Host-Based FailoverMany operating system vendors and other cluster software vendors offer high availability application failover products. These failover solutions monitor application services on a given primary cluster node. They then fail over such services to a secondary cluster node as needed. Host-based failover solutions generally have one active instance performing useful work for a given database application. The secondary node monitors the application service on the primary node and initiates failover when the primary node service is unavailable.Failover in host-based systems involve steps similar to the following:

1. Detecting failure by monitoring the heartbeat2. Reorganizing cluster membership in the Cluster Manager3. Transferring disk ownership from the primary node to a secondary node4. Restarting application and database binaries (Oracle executables)5. Performing application and database recovery6. Reestablishing client connections to the failover node.

You can take leverage this type of configuration using Real Application Clusters Guard, a much more resilient failover option for Oracle9i which is discussed in the next lesson.



Real Application Clusters Failover

• Detect failure by monitoring the heartbeat• Reorganize instance membership• Perform instance recovery• Reestablish failed client connections

Real Application Clusters FailoverReal Application Clusters provide rapid server-side failover. This is accomplished by the concurrent, active-active architecture in Real Application Clusters. In other words, multiple Oracle instances are concurrently active on multiple nodes and these instances synchronize access to the same database. All nodes also have concurrent ownership and access to all disks. When one node fails, all other nodes in the cluster maintain access to all the disks; there is no disk ownership to transfer, and database application binaries are already loaded into memory.Depending on the size of the database, the duration of failover can vary. The larger the database, or the greater the size of its data files, the greater the failover benefit in using Real Application Clusters.You can use a callback function through an OCI call to notify clients of the failover so that the clients do not misinterpret the delay for a failure. This prevents the clients from manually attempting to reestablish their connections.



TAF in Real Application Clusters

• Can be used with spare nodes or with primary/secondary instance configurations

• Designed for this environment, but can be used for – Real Application Clusters Guard– Replicated systems– Data Guard

TAF in Real Application ClustersThe transparent application failover (TAF) feature automatically reconnects applications to the database if the connection fails. Because the re-connection happens automatically within the OCI library, you do not need to change the client application to use TAF.Because most TAF functionality is implemented in the client-side network libraries (OCI), the client must use the Oracle Net OCI libraries to take advantage of TAF functionality. Therefore, to implement TAF in Real Application Clusters, make sure you use JDBC OCI instead of PL/SQL packages.You can also use TAF in Primary/Secondary Instance configurations. If you do this, then you must use the INSTANCE_ROLE parameter (discussed later) in the Connect Data portion of the connect descriptor to configure explicit secondary instance connections.To use TAF, you must have a license for the Oracle9i Enterprise Edition. Because TAF was designed for Real Application Clusters, it is much easier to configure TAF for that environment. However, TAF is not restricted for use with Real Application Clusters environments. You can also use TAF for single instance Oracle. In addition, you can also use TAF for Oracle Real Application Clusters Guard, Replicated systems and Data Guard.



TAF Failover Mode Options

• Must add failover options manually to TNS configuration files

• They are part of the CONNECT_DATA section of a connect descriptor

• Failover options include– TYPE: Identify the nature of TAF, if any– METHOD: Configure how quickly failover can occur– BACKUP: Identify an alternate net service name– RETRIES: Limit the number of times a reconnection

will be attempted– DELAY: Specify how long to wait between reconnection

attempts

TAF Failover Mode OptionsTo implement TAF, you need to include a FAILOVER_MODE parameter in the CONNECT_DATA section of a connect descriptor. If your connect descriptors are defined in TNS configuration files, you will need to add the TAF parameters manually. This is because Oracle Net Manager does not provide support for TAF configurations.The FAILOVER_MODE parameter has a set of subparameters that control how a failover will occur should a client be disconnected from the original connection made with the connect descriptor. The subparameters, which are covered in detail on the following pages, include:•TYPE: (Required) Specify one of the three types of Oracle Net failover available by

default to OCI applications.•METHOD: Determines how fast failover occurs from the primary node to the backup

node.•BACKUP: Specify a different net service name for backup connections. •RETRIES: Specify the number of times to attempt to connect after a failover.•DELAY: Specify the amount of time in seconds to wait between connect attempts.



• Failover types identify the nature of TAF, if any• The options are:

– SESSION: Failover to an alternate session only– SELECT: Failover and continue with any ongoing query– NONE: Prevent failover

Failover Types

. . .(CONNECT_DATA = (SERVICE_NAME = rac.us.aaacme.com)(FAILOVER_MODE =(TYPE=SELECT)

. . .

Failover TypesThree types of Oracle Net failover functionality are available by default to OCI applications:•SESSION: Causes a failed session to failover to a new session. If a user's connection is

lost, a new session is automatically created for the user. This type of failover does not attempt to perform any actions after connecting the user to the new session. This option is your best choice for applications that primarily perform DML transactions and short queries.

•SELECT: Causes a failed session to failover to a new session and continue any interrupted queries. After automatically connecting the user to a backup session, this option enable users with open cursors to continue fetching on them after failure. However, this mode involves overhead on the client side in normal select operations. You should use this option when an instance failure could result in having to recreate output already generated by a long-running query.

•NONE: No failover functionality is used. Although this is the default, you can specify this type explicitly to prevent failover from happening. This option is typically useful for testing purposes rather than for implementing failover in a production environment.



Failover Methods

• Determine how quickly connections become available following a failover

• BASIC: Establishes no contact with the failover instance prior to failure

• PRECONNECT: Creates mirror connections on the standby instance for the connections on the primary instance

. . .(CONNECT_DATA = (SERVICE_NAME = rac.us.aaacme.com)(FAILOVER_MODE =(METHOD=PRECONNECT)

. . .

Failover MethodsThe METHOD subparameter takes one of two values: BASIC and PRECONNECT. The latter is only of use with Real Application Clusters (unlike the TYPE options which can be used for other failover situations, such as standby databases or reconnections to the same instance). The BASIC option requires a session to make a new connection when it fails over from its original instance connection. This option causes no overhead on the backup instance until a failover occurs. This allows you to use the backup instance for nonapplication work, such as database maintenance, without impacting the failover status. However, the failover processing can be slow because all of the disconnected sessions will attempt to reconnect to the failover instance concurrently, overburdening the listener on that instance.The PRECONNECT option provides faster failover by creating a failover connection on the standby instance concurrently with each connection to the primary instance. When the primary instance fails, the connections are switched to one of the existing connections on the standby instance. This requires minimal work by listener for that instance and avoids the overhead of creating new session connections. Unlike the BASIC option, the PRECONNECToption imposes a load on the standby instance which must be able to support all connections from every supported instance.



Failover Backup Service

• Specifies the net service name where failed connections will be directed

• Allows you to identify a specific backup instance connect descriptor for each primary instance

• Use with the PRECONNECT method to identify the service to initiate your failover preconnections

. . .(CONNECT_DATA = (SERVICE_NAME = rac.us.aaacme.com) (FAILOVER_MODE =(METHOD=PRECONNECT)(BACKUP=rac2.us.aaacme.com)

. . .

Failover Backup ServiceThe BACKUP subparameter specifies the net service name to which backup connections should be directed. This allows you to identify a specific backup instance for each primary instance or, if you have only one spare instance, to ensure that failed over connections are all directed to that instance.If you are using the PRECONNECT method for failover, you must use this option to identify the service where you want to start your preestablished failover connections. This same service will, of course, be used when connections fail over.



Failover Connection Reattempt Options

• RETRIES: The number of times to attempt to connect after a failover

• DELAY: The amount of time in seconds to wait between connection attempts

• Set the values based on your failover strategy and test results

. . .(CONNECT_DATA = (SERVICE_NAME = rac.us.aaacme.com) (FAILOVER_MODE =(RETRIES=20)(DELAY=15)

. . .

Failover Connection Reattempt OptionsThere are two subparameters that control how often and how frequently a session will attempt to make a new connection, RETRIES and DELAY. In a Real Application Clusters database with one more standby instances ready and available to support failover activity, connections should made quickly so these parameters can have low values. However, you need to allow time for the listeners on the failover nodes to handle all the reconnection requests from the failed instance. This will be longer if you are not using the preconnection method.These parameters need higher values if the failover is to be made to the same instance (typically for a nonclustered database) or to a standby database because the required service is not immediately available following the initial failure.The RETRIES subparameter specifies the number of times to attempt to connect after a failover. If DELAY is specified, RETRIES defaults to five retry attempts.The DELAY subparameter specifies the amount of time in seconds to wait between connection attempts. If RETRIES is specified, DELAY defaults to one second.You should confirm that you defined appropriate values for these parameters by testing your failover strategy.



TAF Implementation

The recommended configurations for TAF include:• Connect-time failover with client load balancing• Retrying connections• Pre-establishing connections

TAF ImplementationThe FAILOVER_MODE subparameter values can be combined in many ways. This allows you to implement a wide variety of TAF strategies. Oracle recommends that you configure TAF with one of the following methods for the best results:

• Connect-time failover combined with client load balancing• Retrying connections• Preestablishing connections.



Connect-Time Failover with Client Load Balancing

rac.us.aaacme.com=(DESCRIPTION=(LOAD_BALANCE=on)(FAILOVER=on)(ADDRESS=(PROTOCOL=tcp)(HOST=aaacme1)(PORT=1521))

(ADDRESS=(PROTOCOL=tcp)(HOST=aaacme2)(PORT=1521))

(CONNECT_DATA=(SERVICE_NAME=rac.us.aaacme.com)(FAILOVER_MODE=(TYPE=select)(METHOD=basic))))

Connect-Time Failover with Client Load BalancingImplement TAF with connect-time failover and client load balancing for multiple addresses. In the example shown, Oracle Net Services connects randomly to one of the protocol addresses on aaacme1 or aaacme2. If the instance fails after the connection, the TAF application fails over to the other node’s listener, reserving any SELECT statements in progress.Note: Do not set the GLOBAL_DBNAME parameter in the SID_LIST_listener_namesection of your listener.ora file. A statically configured global database name disables TAF.



Retrying Failover Connections


(CONNECT_DATA=(SERVICE_NAME=rac1.us.aaacme.com)(FAILOVER_MODE=(TYPE=select)(METHOD=basic)(BACKUP=rac2) (RETRIES=20)(DELAY=15))))

rac2=(DESCRIPTION=. . .

Retrying Failover ConnectionsEnable the automatic ability of TAF to retry if the first connection attempt fails by setting one or both of the RETRIES and DELAY subparameters. In this example, initial connections are made to the listener on node aaacme1 using the connect descriptor defined by the alias RAC1. If that connection fails, Oracle Net Services tries to make a failover connection, preserving any SELECT statements in progress, to the listener defined with the connect descriptor identified with the RAC2 alias. If the failover connection fails, Oracle Net waits 15 seconds before trying to reconnect again. Oracle Net makes 20 attempts to complete a reconnection through the RAC2 alias.



Preestablishing a Connection for TAF


(CONNECT_DATA=(SERVICE_NAME=rac.us.acme.com)(INSTANCE_NAME=rac1)

(FAILOVER_MODE=(BACKUP=rac2)(TYPE=select)(METHOD=preconnect)

)))


(CONNECT_DATA=(SERVICE_NAME=rac.us.acme.com)(INSTANCE_NAME=rac2)

(FAILOVER_MODE=(BACKUP=rac1)(TYPE=select)(METHOD=preconnect)

)))

Preestablishing a Connection for TAFA backup connection can be preestablished. The initial and backup connections must be explicitly specified as shown in the example of a tnsnames.ora file. Using this file, clients that connect through the service name rac1 to connect to the listener on aaacme1 are also preconnected to aaacme2. If aaacme1 fails after the connection, Oracle Net fails over to aaacme2, preserving any SELECT statements in progress. Likewise, Oracle Net preconnects to aaacme1 for those clients that use the alias aaacme2 to connect to the listener on aaacme2.



TAF Verification

SQL> SELECT machine, failover_type,2 failover_method, failed_over, COUNT(*)3 FROM v$session4 GROUP BY machine, failover_type,5 failover_method, failed_over;

MACHINE FAILOVER_TYPE FAILOVER_M FAI COUNT(*)

------- ------------- ---------- --- --------

aaacme1 SELECT BASIC NO 11

aaacme2 NONE NONE NO 1

TAF VerificationTo determine if TAF is correctly configured and that connections are associated with a failover option, you can examine the V$SESSION view. To obtain information about the connected clients and their TAF status, examine the FAILOVER_TYPE, FAILOVER_METHOD, and FAILED_OVER columns. The example includes one query you could execute to verify that you have correctly configured TAF.The output shown indicates how sessions might appear before they have been failed over. Following a failover, the output could be similar to the following (in both cases, the output has been formatted):

MACHINE FAILOVER_TYPE FAILOVER_M FAI COUNT(*)------- ------------- ---------- --- --------aaacme2 SELECT BASIC YES 11aaacme2 NONE NONE NO 1

Note: You can also monitor each step of TAF using an appropriately configured OCI TAF CALLBACK function.



Simple Primary/Secondary Configuration

• The primary/secondary configuration is supported on two-node clusters

• It provides high levels of availability since both instances are active

• Only the primary instance accepts application client connections

• The secondary instance is available for local connections and can be used for – DBA jobs and report generation– Offloading work from the primary enabling more users

• Provides higher levels of availability when combined with TAF

Simple Primary/Secondary ConfigurationThe Real Application Clusters Primary/Secondary configuration, discussed in an earlier lesson, is a starting place for two-node high availability solution. However, once you are comfortable with this approach, you should consider exploiting the full capacity of the technology and move on to the next step, high availability with TAF.



Primary/Secondary Configuration with TAF

. . .*.active_instance_count = 1*.cluster_database_instances = 2rac1.instance_name = rac1rac2.instance_name = rac2. . .

. . .(CONNECT_DATA=

. . .(INSTANCE_ROLE=ANY). . .

. . .

Primary/Secondary Configuration with TAFAs discussed in an earlier lesson, you need to configure the initialization file by setting the ACTIVE_INSTANCE_COUNT parameter value to 1 on both instances to enable primary and secondary instance configuration. This is shown in the first example. You can also include the optional INSTANCE_ROLE parameter in the CONNECT_DATAsection of a connect descriptor, as shown in the second example. It enables you to specify a connection to the primary or secondary instance of a Real Application Clusters database or a Real Application Clusters Guard configurations (discussed in a later lesson).This parameter is useful when:

• You want to explicitly connect to a primary or secondary instance. The default is the primary instance.

• You want to use TAF to preconnect to a secondary instance.INSTANCE_ROLE supports the following values:•PRIMARY: Specifies a connection to the primary instance•SECONDARY: Specifies a connection to the secondary instance•ANY: Specifies a connection to whichever instance has the lowest load, regardless of the

primary or secondary instance role.



Example

RAC1 =(DESCRIPTION=(LOAD_BALANCE=OFF)(FAILOVER=ON)(ADDRESS=(PROTOCOL=TCP)(HOST=aaacme1)(PORT=1521)) (ADDRESS=(PROTOCOL=TCP)(HOST=aaacme2)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=rac.us.acme.com)(INSTANCE_ROLE=PRIMARY)(SERVER=DEDICATED)(FAILOVER_MODE=(BACKUP=RAC2)(TYPE=SESSION)(METHOD=PRECONNECT)(RETRIES=180)(DELAY =5))))

ExampleThe slide contains the definition of the TNS alias (RAC1) for a connect descriptor that could be used for TAF with a primary/secondary instance configuration. The connections made through the RAC1 alias are to dedicated servers because of the SERVER binding value.The RAC1 alias is for the primary instance, as indicated by the INSTANCE_ROLEsubparameter value. The primary instance is running on node aaacme1 because this is the first address listed in the DESCRIPTION clause and client load balancing, which could select either address, is disabled. Failover is enabled with the FAILOVER parameter and the secondary instance is identified with the RAC2 alias in the BACKUP clause (the connect descriptor for RAC2 is shown on the next page). A failed over session would be directed to preestablished failover connections because of the METHOD subparameter setting and would make up to 180 attempts to complete the reconnection, with a five second pause between each attempt.Note: You could use other options, such as shared instead of dedicated servers, or the BASIC rather than the PRECONNECT method, without interfering with the TAF operations.



Example

RAC2 =(DESCRIPTION=(LOAD_BALANCE=OFF)(FAILOVER=ON)(ADDRESS=(PROTOCOL=TCP)(HOST=aaacme2)(PORT=1521)) (ADDRESS=(PROTOCOL=TCP)(HOST=aaacme1)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=rac.us.acme.com)(INSTANCE_ROLE=SECONDARY)(SERVER=DEDICATED)(FAILOVER_MODE=(BACKUP=RAC1)(TYPE=SESSION)(METHOD=PRECONNECT)(RETRIES=180)(DELAY =5))))

Example (continued)This slide contains the definition for the connect descriptor associated with the TNS alias of the secondary instance, RAC2. The connect descriptor values are similar to those for the RAC1 descriptor with the following key differences:

• The first address listed is for the node where the secondary instance is running (aaacme2). This should prevent any connections made directly through this alias because there is no load balancing to redirect the request to the second address.

• The INSTANCE_ROLE value is defined as SECONDARY. This will also prevent connections through the alias unless the primary instance has failed and the instance on aaacme2 has assumed the primary role.

• The BACKUP value is the alias RAC1 so that connections to the instance on aaacme2 can fail back to the instance on aaacme1 if necessary.



Practice Overview: Configuring TAF

This practice covers the following topics:• Configuring TNS parameter files for TAF• Testing TAF configurations



Optional Practice Overview: Configuring TAF

This practice covers configuring TAF with primary/secondary instances.



Summary

In this lesson, you should have learned how to:• Identify the components required to support TAF• Compare TAF configuration options• Implement TAF with a Real Application Clusters

database• Examine TAF status of client connections• Configure a primary/secondary setup with TAF


Backup and Recovery



Objectives

After completing this lesson, you should be able to do the following:• Implement database archiving• Evaluate archive log destinations for multiple

instances• Implement a backup and recovery strategy with

Recovery Manager (RMAN)• Perform operating system backup and recovery

operations



Protecting Against Media Failure

Databasebackups

Mirroreddisks

Archivedlog files

Archivedlog files

Protecting Against Media FailureAlthough Real Application Clusters provides you with methods to avoid or to reduce down time due to a failure of one (or more, but not all) of your instances, you must still protect the database itself which must be shared by all the instances. This means that you need to consider disk backup and recovery strategies for you cluster database just as you would for a nonclustered database.To minimize the potential loss of data due to disk failures, you may want to use disk mirroring technology available from your server or disk vendor. As in nonclustered databases, you can have more than one mirror if your vendor allows it, to help reduce the potential for data loss and to provide you with alternate backup strategies. For example, with your database in ARCHIVELOG mode and with three copies of your disks, you can remove one mirror copy and perform your backup from it while the two remaining mirror copies continue to protect ongoing disk activity. To do this correctly, you must first put the tablespaces into backup mode and then, if required by your cluster or disk vendor, temporarily halt disk operations by issuing the ALTER SYSTEM SUSPEND command. Once the statement completes, you can break the mirror and then resume normal operations by executing the ALTER SYSTEM RESUME command and taking the tablespaces out of backup mode.



Log History Records

SELECT records_total, records_usedFROM v$controlfile_record_sectionWHERE type ='LOG HISTORY'

CREATE DATABASE . . . MAXLOGHISTORY 500 . . .

CREATE CONTROLFILE . . . MAXLOGHISTORY 1000 . . .

RECORDS_TOTAL RECORDS_USED------------- ------------

500 500

Log History RecordsYour Real Application Clusters control file maintains a record of the archived log files generated for each thread of redo. This allows an instance that is performing media recovery to identify the archived log files it needs regardless of which instance generated them. The number of entries maintained in the control file is determined by the value assigned with the MAXLOGHISTORY in your CREATE DATABASE command.As a best practice, MAXLOGHISTORY should be no less than the total number of archived log files that will be generated across all your instances between each complete database backup. This will allow a recovering instance to identify the archived log files it requires after you restore the most recent backup .If there are insufficient archived log entries in your control file, you will be prompted for the required file names when you initiate the recovery process. This can be hard to do and is almost always time consuming. To increase the size of the MAXLOGHISTORY setting, you must recreate your control file.



Initiate Archiving

1. Shut down all instances2. Start up an exclusive instance with

LOG_ARCHIVE_* parameters set appropriately3. Enter the command: ALTER DATABASE

ARCHIVELOG4. Modify the LOG_ARCHIVE_* parameters for each of

the other instances5. Shut down the exclusive instance6. Restart your instances using the modified

parameters

Initiate ArchivingTo enable archive logging, your database must be mounted, but not open, by an exclusive instance. If you are using an SPFILE, you must first create sid-specific entries for this instance, otherwise build a special-purpose text parameter file. The parameters you must set for this exclusive instance include: •CLUSTER_DATABASE: Set to FALSE•LOG_ARCHIVE_DEST_n: Up to ten values, depending on your archive strategy•LOG_ARCHIVE_FORMAT: Include the %t or %T parameter for thread identification•LOG_ARCHIVE_START: Set to TRUE

You can now enable archiving for your database by employing the following steps:1. Shut down all instances2. Start up your exclusive instance 3. Enter the command: ALTER DATABASE ARCHIVELOG4. Modify the LOG_ARCHIVE_* parameters for each of the other instances5. Shut down the exclusive instance and reset its CLUSTER_DATABASE parameter6. Restart all of your instances using the modified parameters.



Archived Log File Configurations

Cluster file system scheme:archive logs from each instance written to same file location

Local archive with NFS scheme: Each instance can read mounted archive destinations of all instances

Archived Log File ConfigurationsDuring backup and recovery operations involving archived log files, Oracle determines the file destinations and names from the control file. The archived log file path names can also be stored in the optional recovery catalog if you are using RMAN. The archived log file path names do not include the node name, however, so RMAN expects to find the files it needs on the nodes where the channels are allocated. If you are using a cluster file system, your instances can all write to the same archive log destination. This is known as the cluster file system scheme. Backup and recovery of the archive logs are easy since they are all located in the same directory. If a cluster file system is not available, then Oracle recommends that local archive log destinations are created for each instance with NFS read mount points to all other instances. This is known as the local archive with NFS scheme. During backup, you can either backup the archive logs from each host or can select one host to perform the backup for all archive logs. During recovery, one instance may access the logs from any host without having to first copy them to its local destination.Using either scheme, you may wish to provide a second archive destination to avoid single points of failure.



Oracle Recovery Manager

RMAN provides the following benefits for Real Application Clusters• Can read cluster files

or raw partitions with no configuration changes

• Can access multiple archive log destinations

Recovery catalog

RecoveryManager

Backup storage

OracleServer

process

Snapshot control file

Oracledatabase

Storedscripts

Archivedlog files

Oracle Recovery ManagerRecovery Manager (RMAN) can use stored scripts, interactive scripts, or an interactive GUI front end. When using RMAN with your Real Application Clusters database, use stored scripts to initiate the backup and recovery processes from the most appropriate node.If you use different Oracle Home locations for your Real Application Instances on each of your nodes, create a snapshot control file in a location that can be accessed by all your instances, either a cluster file or a shared raw device. For example:

CONFIGURE SNAPSHOT CONTROLFILE TO '/oracle/db_files/snaps/snap_prod1.cf';

For recovery, you must ensure that each recovery node can access the archive log files from all instances using one of the archive schemes discussed earlier, or make the archived logs available to the recovering instance by copying them from another location.



Configuring RMAN

• Configure the snapshot control file location in RMAN

• Configure the control file automatic backup feature

Snapshot Control File ManagementRMAN creates snapshot copies of your control file as part of its backup operations. In a Real Application Clusters database, the snapshot control file is created on the node that is making the backup. You need to configure a default path and file name for these snapshot control files that is valid on every node from which you might initiate an RMAN backup. You can use a shared file system location, including a raw device, if you want.

Automatic Control File BackupsRMAN optionally creates control file backups automatically after BACKUP and COPY commands. These backups can be used if the recovery catalog and current control file are lost. RMAN performs the control file autobackup on the first allocated channel. When you allocate multiple channels with different parameters, especially if you allocate a channel with the CONNECT command, you must determine which channel will perform the automatic backup. Always allocate the channel for the connected node first.



User-Managed Backup Methods

• For offline backup, shut down all instances

• Can use multiple nodes for parallel backup

• Can convert tablespace between backup modes from any instance

• Provide access to all threads of archived redo

• Full offline backupsonly

• Recovery to point oflast backup

With-outarch-iving

• Full or partial, offline oronline backups

• Recovery to point offailure

Witharch-iving

User-managed backup

User-Managed Backup MethodsThe user-managed backup and recovery options and methods for Real Application Clusters database are similar to the procedures used in a noncluster database. These include full offline backups and, if you running in ARCHIVELOG mode, online backups.There are some additional issues that you need to consider when backing up your Real Application Clusters database, including:

• You must shut down every instance for an offline database backup• You can employ more than one node to backup different data files in parallel• You need to issue the ALTER TABLESPACE commands (BEGIN BACKUP and END BACKUP) only on one instance and it does not have to be on the same node where you perform the disk backup operations

• You must provide for the backup and recovery operations to access the archive log files that are independently written by each instance.



Offline User-Managed Backup

• Query the following views to find files that need to be backed up:– V$DATAFILE or DBA_DATA_FILES– V$LOGFILE– V$CONTROLFILE– V$PARAMETER

• Shut down all instances• Copy the identified files to a backup destination• Partial offline backups are performed in exactly the

same way whether the database is clustered or single-instance

Offline User-Managed BackupThe full offline user-managed backup procedure for a Real Application Clusters database is almost identical to the procedure for a single-instance configuration. The only difference is that you must shut down all of the instances, not just one, before you begin the actual backup. The main steps in the backup procedure include:

1. Query the V$DATAFILE view to obtain the names and locations of the data. 2. Query the V$LOGFILE view to obtain the names and locations of the online redo log

files.3. Query the V$CONTROLFILE view to obtain the names and locations of the control

files.4. Query the V$PARAMETER view to obtain the name of the SPFILE, if one is used.5. Shut down all instances that are currently accessing the database.6. After shutting down all instances, use an operating system utility to save all the data

files, online redo log files, at least one copy of the control file, and the SPFILE.7. Restart the instances.

The easiest way to perform this type of backup is to use a script. However, the script must confirm that all instances are shut down before starting to copy any of the files.



Online User-Managed Backup

• ARCHIVELOG mode is required• Perform these steps for each tablespace to be

backed up– Execute ALTER TABLESPACE…BEGIN BACKUP– Invoke the operating system utility for file backup– Execute ALTER TABLESPACE…END BACKUP

• Execute ALTER DATABASE BACKUP CONTROLFILE TO filename or ALTER DATABASE BACKUP CONTROLFILE TO TRACE

• Execute ALTER SYSTEM ARCHIVE LOG CURRENTand backup archived log files from all threads

Online User-Managed BackupOnline backups enable you to back up all or part of the database while it is running. The procedure is essentially the same as for performing an online backup of a nonclustered database. Online backups can be performed only when the database is running in ARCHIVELOG mode. The main steps in the backup procedure include:

1. Execute the ALTER TABLESPACE…BEGIN BACKUP command. This should be done for each tablespace before you back up any of its data files. You can set all tablespaces in backup mode at once, or individually if you backup one at a time.

2. Issue the operating system commands to back up the data files for the tablespace.3. Execute the ALTER TABLESPACE…END BACKUP command.4. Back up the control files with the ALTER DATABASE BACKUP CONTROLFILE

TO filename command. For an additional safety, back up the control file to a trace file with the ALTER DATABASE BACKUP CONTROLFILE TO TRACE command.

5. Execute an ALTER SYSTEM ARCHIVE LOG CURRENT command to archive all redo threads, including any unarchived logs from closed threads, and back up these along with any other archive log files that have not been previously backed up.



Restoring and Recovering Redo Log Files

• Media recovery may require one or more archived log files from each thread

• The RMAN RECOVER command automatically restores and applies the required archived logs

• Archive logs may be restored to any node performing the restore and recover operation

• Logs must be readable from the node performing the restore and recovery activity

• Recovery processes request additional threads enabled during the recovery period

• Recovery processes notify you of threads no longer needed because they were disabled

Restoring and Recovering Redo Log FilesMedia recovery of a database accessed by Real Application Clusters may require at least one archived log file for each thread. However, if a thread’s online redo log contains enough recovery information, restoring archived log files for any thread is unnecessary.If you use RMAN for media recovery and you share archive log directories, you can change the destination of the automatic restoration of archive logs with the SET clause to restore the files to a local directory of the node from where you begin recovery. If you backed up the archive logs from each node without using a central media management system, you must first restore all the log files from the remote nodes and move them to the host from which you will start recovery with RMAN. However, if you backed up each node’s log files using a central media management system, you can use the RMAN SET AUTOLOCATEcommand. This enables you to recover a database using the local tape drive on the remote node:If recovery reaches a time when an additional thread was enabled, the recovery process requests the archived log file for that thread. If you are using a backup control file, when all archive log files are exhausted you may need to redirect the recovery process to the online redo log files to complete recovery. If recovery reaches a time when a thread was disabled, the process informs you that the log file for that thread is no longer needed.



Parallel Recovery in Real Application Clusters

• The RECOVERY_PARALLELISM initialization parameter – Specifies the number of redo application server

processes participating in instance or media recovery – Cannot exceed the value of the PARALLEL_MAX_SERVERS parameter

– Causes serial recovery if the value is zero or one• Parallel recovery can also be initiated with the PARALLEL clause of the RECOVER command

Parallel Recovery in Real Application ClustersParallel recovery uses multiple CPUs and I/O parallelism to reduce the time required to perform thread or media recovery. Parallel recovery is most effective at reducing recovery time while concurrently recovering several data files on several disks. You can use parallel instance and crash recovery as well as parallel media recovery in Real Application Clusters. You can use RMAN or other Oracle tools, such as SQL*Plus, to perform parallel recovery in Real Application Clusters.With RMAN’s RESTORE and RECOVER statements, Oracle automatically parallelizes recovery:

• Restoring data files: the number of channels allocated determines the maximum degree of parallelism

• Applying incremental backups: the maximum degree of parallelism is determined by the number of allocated channels

To perform parallel recovery with other tools, either set the RECOVERY_PARALLELISMinitialization parameter or include the PARALLEL clause in your RECOVER command.



Parallel Recovery in Real Application Clusters

• Parallel recovery uses one process to read the log files and multiple processes to apply redo to the data files

• Oracle automatically starts the recovery processes—you do not need to use more than one session

• The processes may be on one instance or spread across all instances



Fast-Start Parallel Rollback in Real Application Clusters

SQL> DESCRIBE gv$fast_start_transactionsName Null? Type-------------------------- -------- ---------------INST_ID NUMBERUSN NUMBERSLT NUMBERSEQ NUMBERSTATE VARCHAR2(16)UNDOBLOCKSDONE NUMBERUNDOBLOCKSTOTAL NUMBERPID NUMBERCPUTIME NUMBERPARENTUSN NUMBERPARENTSLT NUMBERPARENTSEQ NUMBER

Fast-Start Parallel Rollback in Real Application ClustersIf you have long-running transactions and your nodes have multiple CPUs, you should consider setting the initialization parameter FAST_START_PARALLEL_ROLLBACK to a nondefault value. This allows the rollback step of the recovery process to be performed in parallel. Since each instance is responsible for its own recovery processes, you need to monitor and tune the parallel rollback using the contents of V$FAST_START_SERVERSand V$FAST_START_TRANSACTIONS on each instance or the global views, GV$FAST_START_SERVERS and GV$FAST_START_TRANSACTIONS, from one of the active instances. Although fast-start parallel rollback does not perform rollback activity across instances, it can improve the processing of rollback segments for a Real Application Clusters database.



Practice Overview: Backing Up and Recovering Data

This practice covers the following topics:• Archiving with multiple destinations for each

instance• Performing a backup and recovery with RMAN



Summary

In this lesson, you should have learned how to:• Initiate archiving for the database• Plan an archive strategy for RMAN• Perform backup and recovery with RMAN and

operating system commands• Set parameters to help reduce recovery times


Online Transaction Processing Considerations



Objectives

After completing this lesson, you should be able to do the following:• Evaluate and configure tablespace storage

management options• Evaluate and configure free space management

options• Manage sequence generators• Avoid contention on index leaf blocks• Implement advanced queuing



Extent Management Options

UET$ FET$

Dictionary management is controlled by the UET$ and FET$ tables

Local management is controlled by bit maps in thedata files

Extent Management OptionsWhen creating tablespaces, other than the SYSTEM tablespace, you need to decide how you want extent allocation and deallocation to be managed:

• By bitmaps stored in the tablespace data files themselves (local management)• By the data dictionary's used extent map table, UET$, and free extent map table, FET$

(dictionary management)



Locally Managed Tablespaces

Locally managed tablespaces • Are recommended by Oracle Corporation• Avoid contention between instances for a small

number of blocks (in UET$ and FET$) during extent management

• Remove fragmentation potential when differently-sized extents share a tablespace– Reduce need to defragment free space– Allow you to group segments used primarily by single

instances, enabling dynamic resource remastering• Enable automatic segment free space management

Locally Managed Tablespaces Oracle recommends that you use local management for all new tablespaces and that you consider converting any dictionary managed tablespaces to local management if possible. This is because access to the FET$ and UET$ tables can become a bottleneck, particularly when instances perform concurrent space management operations and need to update these tables.In addition to reducing inter-instance contention, locally managed tablespaces simplify your database design because you can mix segments of different sizes but similarly functionality in the same tablespace without incurring problematic free space fragmentation. It can be useful to combine segments in a single tablespace if some of your instances are used to perform different functions. This is because it allows the Global Cache Service dynamically to migrate the data file resources to the instance where the most of work is being performed, thereby gradually reducing interconnect traffic. This may be less important if you have inherited a database that has been functionally partitioned by instance because such a database should already contain instance-specific tablespaces managed by multiblock locks.



Dictionary Managed Tablespaces

May exist to support an application partitioned under an earlier Oracle release, in which case• Determine if tablespace layout is still valid• Evaluate the use of Cache Fusion compared to

retaining the GC_FILES_TO_LOCKS settings

Dictionary Managed TablespacesIf you have inherited a database that has been partitioned so that each instance works primarily with its own subset of tables, you may decide to continue the use of dictionary managed tablespaces. Although locally managed tablespaces are recommended, if you have an existing database with dictionary managed tablespaces, you will achieve acceptable performance in many cases. Performance will be mainly determined by whether a tablespace has lots of space management operations, that is, extent allocation and deallocation.Even if you continue to use dictionary managed tablespaces, your performance could benefit from employing Cache Fusion. You may wish to compare performance with and without your current GC_FILES_TO_LOCKS initialization parameter settings after you upgrade from an earlier release.



Free Block Management

Bitmap blocks control access to

data blocks in automatic segment-space management

Header block contains free list pointers to first

data block in each set of linked free blocks

Free Block ManagementThere are two methods to locate blocks allocated to a segment that contain free space: automatic space management and free list access. Automatic space management only works for segments stored in locally managed tablespaces and simplifies the design and management of your database. Automatically managed space segments typically reduce the number of blocks that need to be shared between instances without requiring additional work on your part.Free list managed segments can be used in dictionary or in locally managed tablespaces. To avoid contention between instances that could require free blocks in a segment, you need to build it with groups of free lists. Ideally, you should provide one free list group for each instance. If free space cannot be found using the method defined for the segment, the high-water mark is moved to make more blocks available. This may require the addition of a new extent.



Automatic Segment-Space Management

• Bitmap blocks are stored throughout a segment using automatic space management

• Each bitmap block contains space availability information for a distinct subset of blocks

• Only boundary condition changes in a block's free space availability are recorded– Only one bit needs to be changed to record a change– These changes are fast and cause little contention

• Bitmap blocks are allocated to a session requiring free space based on the– Instance number to avoid inter-instance contention– Session ID to avoid inter-session contention

Automatic Segment-space ManagementAutomatic segment-space management objects use bitmap blocks (BMBs) whose contentsidentify the free space status of a set of blocks with just a few bits. The BMBs are logically organized within a segment for fast retrieval and management of disjoint sets of data blocks. When a session needs to find a block with available free space, it is directed to a BMB using a hash value that is derived from the session number and the ID number of the instance where the session is running. This reduces the likelihood of two sessions needing to use the same BMB and the blocks it manages. Because the instance number is part of the hashing algorithm, there is an even smaller probability that sessions from different instances will be directed to the same blocks.



Enable Automatic Segment-Space Management

CREATE TABLE promo_dates

( promo_id NUMBER . . . )

PCTFREE 5

STORAGE (INITIAL 5M NEXT 5M PCTINCREASE 0)

TABLESPACE promotions;

CREATE TABLESPACE promotions

DATAFILE '/ora/partition_links/promo' SIZE 500M

EXTENT MANAGEMENT LOCAL AUTOALLOCATE

SEGMENT SPACE MANAGEMENT AUTO;

Enable Automatic Segment-space ManagementTo create segments that use automatic segment-space management, you must place them in a tablespace defined with the EXTENT MANAGEMENT LOCAL and SEGMENT SPACE MANAGEMENT AUTO clauses. The tablespace PROMOTIONS, shown in the example, is defined with these clauses.The table PROMO_DATES will be created with automatic segment-space management because it is store in the PROMOTIONS tablespace. Tables with automatic segment-space management cannot be defined with free lists, free list groups, or PCTUSED values.



Free List Space Management

• Free lists consist of blocks with free space that are logically connected with pointers

• The first entry in a free list pointer chain is stored in the segment header block

• Changes to a free list may require updates to the pointer in the header block

• Multiple instances requiring free space in a segment must obtain the current copy of the header block– Sessions need the contents to start a search for free

blocks– Sessions need exclusive access if they have to update

the first pointer in a free list

Free List Space ManagementSegments that use free list space management contain free space information in the segment header block. The header block contains a pointer to the first block on each free list. Each free list consists of one or more blocks that are linked by pointers in the block headers. When a session needs a free block, it will follow the pointer chain for one of the free lists until it finds a block with sufficient space. As new blocks are added to, or full blocks removed from, a free list, the segment header has to be updated. This can cause contention when multiple instances perform operations that frequently need to update the free list entries in the segment header block.



Free listgroup 1header

Segmentheader


Free List Groups

Blocks for generic free list

Blocks for free list group 1

Blocks for free list group 2

Free List GroupsTo reduce the contention for the segment header block by multiple instances requiring access to free lists, you can use free list groups. When a segment is created with multiple free list groups, each group, which can contain many free lists, has its own header block. These special header blocks, rather than the main segment header block, contain a pointer to the first block in each free list associated with the group. For blocks not assigned to a specific free list group, there is a generic group in the segment header block. When an instance needs a free block, it will search for entries on the header block defined through the parameter INSTANCE_NUMBER. If there are as many, or more, free list groups as instances, then instances will not have to share free list group header blocks. The slide shows a segment with two free list groups. Each group has its own header block, and the free lists mastered by each group point to a different set of blocks. This means that each instance can make changes to the free list entries on its own header block without contending with the other instances. Further, the blocks on the free list for any group will be used exclusively by the instance assigned to that group. This will also reduce contention between the instances for access to the data blocks. Note that the graphic only shows one free list in each group.



Manual Allocation of Free List Groups to Instances

CREATE TABLE departments

( department_id NUMBER(6)

, name VARCHAR2(30)

, location_id NUMBER(4))

STORAGE (INITIAL 20K NEXT 100K

PCTINCREASE 0 MAXEXTENTS 5

FREELIST GROUPS 4 FREELISTS 8);

ALTER TABLE departments

ALLOCATE EXTENT (

DATAFILE '\\.\rac_user01'

INSTANCE 1);

Manual Allocation of Free List Groups to InstancesUse the FREELIST GROUPS parameter of the STORAGE clause in a CREATE command to assign multiple free lists to a segment. The command in the first example creates the table DEPARTMENTS with four free list groups, each with eight free lists. Blocks can be assigned to the free lists in one of these groups, and thus to instance using the group, using manual or dynamic allocation. Manual allocation allows you to define which instance will be able to look for free blocks in each extent in the segment. To do this, create extents with an ALTER…ALLOCATE EXTENT command containing an INSTANCE parameter. The second example shows an example for the DEPARTMENTS table created in the first example. In this case, blocks in the new extent will be managed by free lists in the first free list group. The instance with its INSTANCE_NUMBER initialization parameter set to the value 1 will use this free list group and, therefore, the blocks in this new extent. Note that if you start more than four concurrent instances (the number of free list groups assigned to the table), a hashing algorithm may assign additional instances to the first free list group.Note that MAXEXTENTS is set to FREELIST GROUPS + 1 in these examples. This prevents automatic allocation of extents managed by the generic free list group. This should be avoided because instances share the header and data blocks, causing contention.



Automatic Allocation of Free List Groups to Instances

File 10

GC_FILES_TO_LOCKS="10:20!5"Free listgroup 1header

Segmentheader


OldHWM HWM moved by session from instance 1

NewHWM

Blocks added to free list group 1

Automatic Allocation of Free List Groups to InstancesTo avoid the management issues associated with manual allocation of instances to free list groups, you can configure automatic allocation. Automatic allocation works by associating blocks with a free list group at the time they become available as free blocks. This occurs when

• A DML command removes sufficient data from a block to reduce the used space below the PCTUSED threshold, or

• The segment high-water mark (HWM) is moved to make more blocks available.The session that causes the blocks to become free assigns the blocks to a free list in its associated free list group header block. You do not have to do anything to activate automatic free list group allocation once you have defined a table with multiple free list groups. However, if you are not using Cache Fusion on the data files where the table extents are stored, you may wish to assign grouped multiblock cache resources to these data files. Do this with the grouping factor syntax in the GC_FILES_TO_LOCKS initialization parameter as shown in the example. In this case,any session that has to move the HWM for a table in the data file with ID number 10 will not only place the newly-acquired blocks in the free list group associated with that session's instance number, but also assign a single multiblock lock to those blocks.



Comparison of Free Space Management Methods

Key Benefit

Virtually no additional work once created

Can control extent location as well as instance access

Easier free list group management than manual method

Method

Automatic segment-space management

Manual free list group assignment

Automatic free list group assignment

Main Drawback

Upgraded database may require conversion to locally-managed tablespaces

Requires constant monitoring to avoid out-of-space errors

No control over which instances acquire blocks

Comparison of Free Space Management MethodsThe table shows the primary benefit of, and the main drawback to, each space management method in a Real Application Clusters database. Oracle recommends automatic segment-space management for simplicity of space management with minimalcontention between instances for blocks.If you are responsible for a Real Application Clusters database that does not use automatic segment-space management, you may want to recreate the tablespaces and segments to enable this feature. Although this is a manual task, you will almost certainly gain the benefits of better performance and less management overhead after undertaking this activity.



Sequential Numbers

InstanceA

InstanceB

InstanceC

InstanceD

Table that providessequential numbers

Sequential NumbersIf you use a table with a single row to provide sequential numbering, extracting a number requires that you lock the table while the number is read and replaced with the next number in the sequence. This type of transaction serializes access to the sequence numbers and causes multiple block transfers if each instance needs to obtain numbers from the table. Oracle's sequence generators can remove this bottleneck and reduce the frequency of block transfers through the use of caching. This allows applications to scale and you should employ Oracle's sequence generators when you need a sequence number that needs to be shared by multiple instances.



Sequence Generators

• Defined with the SQL CREATE SEQUENCE statement• Real Application Clusters allows multiple instances

to obtain unique integers from the same sequence generator

• The Global Cache Service coordinates sequences across instances

• Sequence generators in Real Application Clusters – Have the same options as single instance databases– Do not work in exactly the same way in all cases

Sequence GeneratorsThe SQL statement CREATE SEQUENCE establishes a database object from which multiple users can generate unique integers without waiting for other users of the sequence to commit their transactions. Users on multiple instances can obtain sequence numbers with minimum cooperation or contention among those instances. Sequences are coordinated by the Global Cache Service across instances in Real Application Clusters.Sequence numbers are always unique, unless the CYCLE option is used but numbers can be assigned out of order if the CACHE option is used without the ORDER option when defining the sequence.



Sequence Generator Options

• The CACHE option – Caches sequence numbers in memory– Improves performance– May lose numbers in the sequence at shutdown

• The ORDER option– Guarantees the order of the requests– Increases overhead

• When both CACHE and ORDER options are defined – Caching is not performed in Real Application Clusters

instances– No error or warning messages indicate that ORDER

takes precedence

Sequence Generator Options The CACHE option of the CREATE SEQUENCE command enables instances to prefetch sequence numbers and retain them in memory for faster access. The number of sequence values to be cached can be specified as an argument to the CACHE option. The default value is 20.Each instance keeps its own cache of sequence numbers in memory. This reduces the need for instances to access and update the data dictionary sequence object. However, cached sequence values are usually lost when an instance shuts down.The ORDER option guarantees that the sequence numbers are delivered in the order of the requests. If you do not need the sequence numbers to be ordered, using the default NOORDER option significantly reduces the overhead for your database.Real Application Clusters does not support the use of the CACHE option with the ORDERoption although you can define both. In this case, the sequence numbers will be ordered but not cached. This is because an instance cannot guarantee extracting cached values from each other's memories in strict chronological order. Every number has to be fetched directly from the sequence object in the data dictionary, with associated Global Cache Service overhead and serialization of the processing.



Index Leaf Block Contention

Rootnode

Branchblocks

Leafblocks

InstanceA

101–120cached

InstanceB

121–140cached

111112125

Index Leaf Block ContentionIndexes are commonly used in an Oracle database to support the primary key on a table. Very often, this is a unique numeric value, provided through a sequence generator, as discussed in the previous few pages. Unfortunately, there is a side effect to using a sequence generator to provide unique primary key values in a Real Application Clusters database. The primary key values generated from each instance tend to be close in values to those generated by the other instances. The instances therefore need to insert the related index records into the same leaf block of the B*-tree. If you have high rates of record inserts across multiple instances, the logical right-most leaf block of the primary key index is constantly transferred between the instances. If you have many tables with this type of index, you may begin to overload the cluster interconnect.



Index Leaf Block Contention

To reduce contention:• Restrict changes to a single instance• Partition tables and underlying index• Use instance-specific sequence generators• Use a multiplier to create distinct ranges• Consider reverse-key indexes

Index Leaf Block ContentionIndex contention in a Real Application Clusters environment can be reduced by one or more of the following techniques:

• Localize all read and write access to this index to a single instance by routing transactions that alter this table to a single instance and running the system asymmetrically.

• Partition the table and use a data-dependent routing strategy.• Use a different sequence generator for each instance, with each generator supplying a

distinct range of values. For example, create the generator for Instance A with a range of 1 to 1,000,000 and the generator for Instance B with a range of 1,000,001 to 2,000,000.

• Use a multiplier to create different ranges of values by instance:Primary key = PROD_SEQUENCE.NEXTVAL + INSTANCE_NUMBER ×1,000,000,000

• If the rate at which new entries are being generated is not too high, and you are unlikely to need index range scan access to the key columns, reverse-key indexes can help reduce contention on the index leaf blocks



Advanced Queuing Considerations

• Queue table instance affinity• Global Cache Service resource acquisition• Advanced Queuing and queue table cache transfers

Advanced Queuing ConsiderationsAdvanced Queuing provides a robust communication tool for a user session to send messages to another session that may or may not be currently active. A number of the mechanisms that support Advanced Queuing are instance resident. Consequently, Advanced Queuing in Real Application Clusters environments introduces a number of functionality and performance-related issues that are not encountered in single-instance databases:

• Queue table instance affinity• Global Cache Service resource acquisition• Advanced queuing and queue table cache transfers



Queue Table Instance Affinity

SELECT queue_table, primary_instance,

secondary_instance, owner_instance

FROM dba_queue_tables

QUEUE_TABLE PRIMARY_IN SECONDARY_ OWNER_INST

----------- ---------- ---------- ----------

PURCHASE_Q1 1 2 1

PURCHASE_Q2 1 2 1

SALES_Q1 1 2 1

Queue Table Instance AffinityQueue table instance affinity enables you to assign primary and secondary instance properties to queue tables. This allows automatic assignment of queue table ownership during instance reconfigurations. The instance assigned as the primary will be the queue table's owner if it is running, otherwise the secondary instance will become the queue table's owner. To enable this feature, you must assign a queue process to each instance with the instance’s AQ_TM_PROCESSES initialization parameter.You can determine which instances, if any, are assigned as the primary and secondary functions for a queue by querying the DBA_QUEUE_TABLES view. The PRIMARY_INSTANCE and SECONDARY_INSTANCE columns contain the ID numbers for the respective instances. A value of zero in these columns indicates that queue ownership functionality has not been assigned.The OWNER_INSTANCE column of DBA_QUEUE_TABLES contains the instance number of the queue's current owner.



Reduce Queue Table Cache Service Resource Requests

• Avoid unnecessary COMMITs • Use automatic segment-space management for

queue tables and their indexes• Partition applications and queue tables by instance

Reduce Queue Table Cache Service Resource RequestsIn general, cache transfers of queue table data blocks and queue table index blocks occur when:

• Queues are accessed simultaneously from different instances• Oracle incorrectly assigns queue table ownership so that a queue monitor schedules a

queue from an instance that is different from the instance where the enqueue or dequeue operations are performed

• Oracle must perform space transactions on the queue tableGlobal Cache Service resource acquisition is more expensive than local resource or local enqueue acquisition. You may need to make some changes to your deployment of Advanced Queuing to reduce its resource control behavior in Real Application Clusters environments. Consider implementing the following to reduce cache block transfers:

• Remove unnecessary COMMITs when transactions involve DML on the queue tables• Employ automatic segment-space management or multiple free lists and free list

groups for the queue tables and indexes• Partition applications to access a queue from only one instance and create queue tables

for each instance



Transaction Processing in Earlier Releases

Reduce block transfers by• Partitioning work across instances by

– Users– Applications– Transactions

• Grouping segments into separate sets of tablespaces based on use– Read only – Written by only one instance– Written by many instances

Transaction Processing in Earlier ReleasesIf you are using an earlier release of Oracle, data block transfers must use disk writes and reads rather than Cache Fusion. High volumes of disk-based, inter-instance block transfers can seriously degrade performance. You may need to consider a partitioning strategy to reduce the number and frequency of such transfers. Most successful partitioning involves a two-prong approach:

• Partition users, applications, or transactions (using a transaction processing monitor in a middle tier) to concentrate the DML activity for each segment on a single instance.

• Organize your segments so that tablespaces do not contain segments used primarily by more than one instance. If segments will need to be processed with DML commands by more than one instance on a regular basis, group these segments into a separate group of tablespaces.

The goal of this approach is to reduce the overhead of multiblock cache resources while minimizing contention for those resources between instances. Tablespaces containing segments that will be modified by just one instance only need to be assigned a few multiblock locks with the GC_FILES_TO_LOCKS parameter. Tablespaces containing segments shared by multiple instances, which you should attempt to minimize, need the default resource allocation which provides single-block resources on demand.



Summary

In this lesson, you should have learned how to:• Configure tablespace storage management to

reduce inter-instance contention• Configure free space management options• Manage sequence generators in Real Application

Clusters• Avoid contention between instances for index leaf

blocks• Implement advanced queuing in multiple instances


Considerations for Online Analytical Processing, DSS, and Data Warehousing



Objectives

After completing this lesson, you should be able to do the following:• Populate a data warehouse• Configure key elements in a data warehouse

– Denormalized tables– Materialized views and summary tables– Indexes– Partitioned segments– Temporary segments

• Implement parallel processing– Automatic parallelism – Manual configuration



Query-Intensive Database Issues

• Query-intensive databases include– Online analytic processing (OLAP) servers– Decision support systems (DSS)– Data warehouses

• Characterized by – Large amounts of data– Extensive query access– Scheduled batch loads to refresh or replace data

• Processing benefits from high amounts of parallelism

Note: In this lesson, the term “data warehouse” refers to any type of query-intensive database.

Query-Intensive Database IssuesMost databases used for online analytic processing, decision support systems, and data warehousing contain large amounts of data that needs to be retrieved as efficiently as possible. Most queries involve significant manipulation of the data in the underlying relational tables, including multiple table joins, large data sorts, and ad hoc queries that retrieve records in ways that cannot always be predicted during table and index design.Queries of this nature can take advantage of Real Application Clusters by running steps in parallel across multiple instances to achieve execution speedup. Even for queries that have very few steps that can be run in parallel, the user load can be distributed across multiple instances with client load balancing, thus reducing contention for memory and CPU cycles on a single node. Query processing requires minimal inter-instance communication because cache resources can be concurrently retained by all the instances in shared mode. Therefore, query-intensive applications offer a high potential for scalability. However, data in such systems has to be refreshed on a periodic basis and you may need to address some of the design issues discussed in the lesson on transaction processing. You may also want to take advantage of Real Application Clusters features to help improve query performance even further by employing some of the methods discussed in this lesson.



Populating the Data Warehouse

• To move the data to the data warehouse server, use: – External tables– UNIX pipes– Mainframe or legacy connection (ESCON and so on)– Third-party packages (Platinum, Prism, and so on)– Transparent gateways– Oracle9i Net Services

• Employ the cluster nodes effectively– Use multiple, or all, instances if the load techniques

avoid contention– Use a single instance dedicated to load processing if

work cannot be shared effectively

Populating the Data WarehousePopulating the warehouse is a very important step in building an efficient data warehouse, but it is often an overlooked or undervalued process. The periodic loads should be the highest priority activity of the warehouse when they occur.Note that using only one node to perform the data loads will reduce the number of resources that have to be shared and thus avoid overhead for the Global Cache Service. However, if your data is being loaded into different data files or, if the table is partitioned, different tablespaces, you can split the work across multiple instances while avoiding resource contention. If you are using locally managed tablespaces and automated segment space management, you should be able to load different rows into the same table in the same data file without incurring detrimental overhead for instance coordination.



Populating the Warehouse

• Allow enough time for double runs in the data load time window in case of problems

• Tools may require batch processing before data loading

• Design smart batch programs that optimize sequencing and prioritizing– Preprocess data on the most suitable server even if

not part of your Real Application Clusters database– Do not wait for all file transfers to complete before

starting table load operations

Populating the Data Warehouse (continued)To avoid inter-instance traffic, you should try to complete your loads while no queries are running against the tables being loaded. This is particularly true if you are using multiblock locks rather than Cache Fusion because the former are likely to cause unnecessary disk writes and additional disk reads.If you have multiple instances available to perform the load processing, use them as effectively as possible. For example, use one node to collect data from the production systems, one node to scrub the data and perform sorts and so on, and another node to load the data when it is ready. These steps can be performed concurrently once the first set of data is ready for loading.Following a failure of some sort during the load, you may need to use more nodes to rerun the load within the required timeframe. Such an action may impact query performance if you have configured your instances for either query or data load activities.



Data Loading

Load

Extract,Prepare

Extract

ScrubScrub

LoadLoadData

Warehouse

OperationalEnvironment

Spider-web dataextraction

PrepareExtract

Data LoadingTo better control your data-loading operation, including extraction from the production systems, preparation and scrubbing, and specific data loading, try to consolidate the work. If your data loading operations look like the “spider web” diagram shown in the graphic, you are probably not using your resources effectively. You almost certainly will not be able to run many of the steps in parallel to reduce the overall load processing time. Further, it can become almost impossible to restart a failed step without having to undo some of all of the work already accomplished.



Data Loading

Extract,prepare,

scrub, load

Organized dataextraction

OperationalEnvironment

DataWarehouse

Extract,prepare,

scrub, load

Extract,prepare,

scrub, load

Data Loading (continued)Centralizing the data-loading operations can simplify your work and enable you to transfer procedures to additional instances more easily if the workload increases. In the diagram, three instances are used to perform the required steps on different files in parallel.Another option would be to perform the extraction on one instance, the preparation on a second instance, scrubbing on another instance, and the load on a fourth instance. This would allow you to “pipeline” the work across the instances, sending a file that has been processed by one instance on to another instance for the next phase.



Denormalization

• Is part of data warehouse database design• Helps with areas of high contention• Aids in reducing repetitive joins• Users should perform further denormalization if

required– Employ temporary tables– Build with parallel create table as select (PCTAS)

operations

DenormalizationPart of the process of moving data from the production database to the warehouse database should involve denormalization. This can be done on an intermediate machine or, if you have a large cluster environment, on nodes reserved specifically for this purpose. Data warehouse users may want to perform further denormalization steps. Encourage them to use temporary tables for intermediate steps or for data they want to reuse but not keep permanently. Recommend that they employ parallel create table as select (PCTAS) commands whenever possible in order to take advantage of available instances for these operations.



Materialized Views

• Materialized views– Store non-normalized data– Pre-join records from a fact and multiple dimension

tables– Should be scanned using parallel queries

• Summary tables– Are materialized views containing summary data– Are based on rolling summarizations, for example, by

week, then month, then quarter– Can be made at the levels most frequently needed by

the clients– Should be the most frequently accessed contents of

the data warehouse

Materialized Views Materialized views reduce the amount of processing needed to execute queries that would normally require CPU intensive operations such as joining multiple tables, sorting data, and performing group operations to summarize data. Once created, materialized views can be queried directly by users or applications that know the materialized view names. However, sessions that have query rewrite enabled will access materialized views automatically when the optimizer transforms the queries against the base tables. Although the materialized views may contain more, non-normalized data than the underlying tables, they contain fewer records and do not require the overhead of the join operations to produce the result set.Summary tables are specialized type of materialized view that contain the detailed data stored in the warehouse compressed into summary records. Depending on the clients' needs, tables can be summarized on different columns. The CREATE DIMENSION command can be used to identify relationships between columns for further summarizations on existing summary tables. Query rewrites can transform a query on one or more base tables to query a summarized table, possibly using dimensions defined on it. Queries against summary tables are almost always more efficient than queries against the base tables because fewer records have to be read and grouped.



Summary Tables

Daily transactions

Daily transactions

Daily transactions

Daily transactions

Daily transactionsWeekly transactionsWeekly transactionsWeekly transactions

Monthly transactionsLess detailed records

Very detailed records

Highly summarized records

Summary TablesSummary tables combine records with similar characteristics into groups and summarize the data for each group. You can provide various summary levels for different users, and each less detailed level can be built from the contents of the more detailed summaries. Most typically, the summary is based on periods of time. This type of summary table is frequently maintained on a rolling schedule. A summary table built using the CREATE MATERIALIZED VIEW command may be accessed when a query against the base table is rewritten automatically by the optimizer.



Summary Tables

CREATE MATERIALIZED VIEW sales_summaryPARALLELBUILD IMMEDIATEREFRESH FASTENABLE QUERY REWRITEAS SELECT /*+ PARALLEL */ t.calendar_month_desc,c.cust_state_province,SUM(s.amount_sold) AS sum_salesFROM times t, sales s, customers cWHERE s.time_id = t.time_idAND s.cust_id = c.cust_idGROUP BY t.calendar_month_desc,c.cust_state_province

Summary Tables (continued)Summary tables are created with the CREATE MATERIALIZED VIEW command as shown in the example. Summary tables reduce the time required to perform queries because they contain fewer rows and may include joined records from fact and dimension tables. Summary levels can range from very basic (light), such as when just one column is used to create the summary table, to very heavy, where virtually no detailed data from the base tables is maintained. Time-based summaries can be replaced on a rolling schedule to keep only the most recent data in the summary.Real Application Clusters can help maintain summary tables efficiently by performing the refresh activities with a high degree of parallelism.



Index Creation

• Use the PARALLEL clause to speed index creation• Presort load data along index boundaries if possible

to enable use of the NOSORT option• Consider bitmap and bitmap join indexes when

planning your index strategy

Index CreationUse the ability of Real Application Clusters to scan and process across instances to build indexes in parallel. Include the PARALLEL clause in the CREATE INDEX command to enable parallel execution of the statement on one or more nodes. If your instances can create indexes quickly, you should consider dropping all indexes prior to data load operations and rebuilding them after the load. This will allow you to set the PCTFREE value to zero and make the indexes more efficient in general.If you are moving the data from a production database to a warehouse database, you may want to sort the data before loading it into the warehouse. This will enable you to use the NOSORT option when building the index on the sorted columns, saving the time and resources needed for a database sort.You may want to build bitmap or bitmap join indexes in place of standard B*-tree indexes for some columns and tables.



Partitioning Option

• Provides an automatic partitioning capability• Enables indexes to be partitioned with the table

partitions or independently of them• Enables the use of parallel DML• Can help avoid inter-instance resource contention

and block transfers during data loads

Partitioning OptionThe Oracle partitioning option is designed to help with the management of very large tables, such as those that occur in data warehouses, by allowing segment management at the partition level rather than on the complete table. In a Real Application Clusters database, partitioning also enables high levels of parallelism for commands that can span multiple instances. Many operations performed by the data warehouse users benefit transparently from partitioning and high degrees of parallelism without any additional work on their part.Queries and DML commands can take advantage of partition pruning as well as parallel manipulation of the data. The work can be confined to parallel execution processes on a single node or be spread across two or more nodes in a Real Application Clusters database.When updating tables by replacing the contents of the oldest partition with new data, use the following strategy to help avoid unnecessary inter-instance traffic during the load process:

• Load the new data into a non-partitioned, working table. Users performing concurrent queries can continue to see the old data without any impact on their performance, particularly if you use instance groups, discussed later, to prevent queries from using the instance performing the load.

• Exchange the oldest partition with the working table, after which you can drop the latter. This is a fast operation that can also take advantage of high levels of parallel execution.



Datamarts

Datamart accessServer

DatamartDatamart

Data warehouse

Datamart access

More datamarts

Server

Standard SQLinterface

Datamartdatabase

Datamartdatabase

Data warehousedatabase

(relational)

DatamartsThe data warehouse can consist of a series of smaller, more easily managed datamarts. Datamarts can help keep the amount of data managed by an individual database at more manageable levels.On a large cluster, the marts can exist on separate sets of nodes. For larger marts, each one may be a separate Real Application Clusters database.



Automatic Control of Parallel Processing

• Set parameters for automatic control of parallel processing – PARALLEL_AUTOMATIC_TUNING– PARALLEL_ADAPTIVE_MULTI_USER

• Use the default degree of parallelism in – Segment definitions– SQL statement hints– DBMS_STATS parameters

Automatic Control of Parallel ProcessingOracle recommends that you allow the instances to determine how they will distribute the workload for parallel operations across the parallel server processes available. Set the initialization parameter PARALLEL_AUTOMATIC_TUNING to TRUE. This will also set the PARALLEL_ADAPTIVE_MULTI_USER initialization parameter, discussed on the next page, to TRUE. Between them, the parameters automatically set values for other parallel server related parameters and enable user processes to determine the number of available instances and processes at execution time. Parallel operations are distributed accordingly.For the automatic parameters to work correctly, the default degree of parallelism should be indicated when defining parallel characteristics, such as:

• The PARALLEL clause in the definition of tables, indexes, clusters, materialized views, and so on. Use the keyword PARALLEL, with no other arguments, where appropriate in the DDL commands used to create or modify these segments.

• The PARALLEL hint in SQL statements. Use only the table name (or table alias) in the hint, not the values for degree or instances. For example,

SELECT /*+ PARALLEL(s) */ COUNT(*) FROM sh.sales s;• The value for the DEGREE parameter in calls to DBMS_STATS procedures. Use the

constant DBMS_STATS.DEFAULT_DEGREE.



Adaptive Multiuser Feature Used in Parallel Execution

• The degree of parallelism specifies the number of available processes, or threads, used in parallel operations

• Use the adaptive multiuser feature to adjust the degree of parallelism

• Apply the adaptive multiuser algorithm

Adaptive Multiuser Feature Used in Parallel Execution The adaptive multiuser feature adjusts the degree of parallelism based on user load. The degree of parallelism specifies the number of available processes, or threads, used in parallel operations. Each parallel thread may use one or two parallel server processes, depending on the statement's complexity. For example, you may have a table against which full scans execute well with a degree of parallelism equal to five. This may be acceptable with up to ten users concurrently performing scans against the table. But if more users enter the system, the instance may need to reduce the degree of parallelism to spread resources more evenly according to the perceived system load. With PARALLEL_ADAPTIVE_MULTI_USER set to TRUE, the instance is able to make such changes as the load changes, based on an algorithm with a number of inputs.With Real Application Clusters, the query execution processes chosen to execute the query will be restricted, if possible, to one node to reduce internode communications. However, if resources are not available on a single node, parallel execution processes from multiple nodes will be used.



Automatic Parallel Execution Optimization

• PARALLEL_EXECUTION_MESSAGE_SIZE • PARALLEL_MAX_SERVERS• PROCESSES• SESSIONS• LARGE_POOL_SIZE

Automatic Parallel Execution OptimizationYou are advised to use automated parallel execution to avoid more intensive administration or user load and system resource miscalculations. You have already seen that, when you do this by setting PARALLEL_AUTOMATIC_TUNING to TRUE, the value of PARALLEL_ADAPTIVE_MULTI_USER is also set to TRUE. Some other initialization parameters also have special values derived for managing parallel execution automatically: •PARALLEL_EXECUTION_MESSAGE_SIZE: set to 4096 bytes on most platforms•PARALLEL_MAX_SERVERS: derived from CPU count, usually ten times the number of

CPUs•PROCESSES: derived from PARALLEL_MAX_SERVERS and CPU count•SESSIONS: derived from PROCESSES, usually(PROCESSES × 1.1) + 5 •LARGE_POOL_SIZE: partially derived from PARALLEL_MAX_SERVERS and PARALLEL_EXECUTION_MESSAGE_SIZE

Although you can still provide your own values in your initialization parameter file, these derived values are usually optimal for most environments and are at least as effective as other settings.Note: After setting PARALLEL_AUTOMATIC_TUNING to TRUE, you can reduce the size of your shared pool by the size automatically added to the large pool.



Manual Control of Parallel Processing

• Set PARALLEL_AUTOMATIC_TUNING to FALSE• Provide values for the parameters that would have

been derived with a setting of TRUE• Define a value for the number of threads of

parallelism in CREATE and ALTER statements, hints, and similar operations

Manual Control of Parallel ProcessingIf you prefer to control all aspects of parallel processing in your Real Application Clusters database, you need to set the PARALLEL_AUTOMATIC_TUNING initialization parameter value to FALSE. In addition, you will need to set the other parameters, discussed previously, that would have had their values derived from this parameter.In addition, you can include a value for DEGREE in SQL statements that allow a PARALLEL clause, as an option in PARALLEL hints, in your calls to DBMS_STATSprocedures, and so on. This will limit the maximum number of parallel threads used by parallel operations. You can still allow operations to execute with a number of parallel threads based on your manual settings by omitting the value for DEGREE where appropriate.



Additional Parallel Processing Parameters

• PARALLEL_MIN_SERVERS – Starts parallel server processes for parallel queries– You are advised to establish a few parallel process

servers if you intend to query the global dynamic performance tables

• PARALLEL_MAX_INSTANCES• PARALLEL_INSTANCE_GROUP• INSTANCE_GROUPS• PARALLEL_BROADCAST_ENABLED

Additional Parallel Processing ParametersOther initialization parameters that influence parallel operations include:•PARALLEL_MIN_SERVERS: Defines the number of parallel execution processes that

will always be available on the instance for use by parallel executions. Note that queries against the global dynamic performance tables (GV$ views) will start and stop parallel execution processes if this parameter has not established a sufficient number.

•PARALLEL_MAX_INSTANCES: Defines the maximum number of instances that will provide parallel server processes for the processing of a single statement. Although this parameter is not dynamic, you can limit parallel processing dynamically to n instances with the command ALTER SYSTEM SET SCAN_INSTANCES=n

•PARALLEL_INSTANCE_GROUP: Identifies a group of instances to which parallel operations will be restricted. Parallel operations will spawn parallel execution processes only on instances in this group.

•INSTANCE_GROUPS: Names one or more instance groups to which this instance belongs. These names are used by PARALLEL_INSTANCE_GROUP values to determine which other instances an instance can use to perform its parallel operations.

•PARALLEL_BROADCAST_ENABLED: Enables the broadcast of rows from a small data set to instances that are joining these rows, in a parallel operation, with a larger data set.



Temporary Tablespaces

• For optimal performance – Use locally managed tempfiles– Spread the files across multiple disks

• Build a single tablespace with many tempfiles to provide most versatile space availability

• Build multiple temporary tablespaces if you require a partitioned user community

Temporary TablespacesYou should anticipate the need for large amounts of temporary space in a Real Application Clusters database used for online analytical processing, decision support systems, or data warehousing. This is because operations on large amounts of data frequently need to sort many rows, for grouping functions, sort merge operations, and so on, as well as ordering the results in a convenient way. Hash join operations may also require temporary space if the HASH_AREA_SIZE parameter cannot be configured large enough. Additionally, someanalysts may make extensive use of temporary tables to store intermediate results to help compare various what-if scenarios. Your temporary space can be allocated in one large tablespace or spread across multiple tablespaces. The former is useful if your database is on a system without shared disks. In this environment, you can place a different tempfile from the temporary tablespace on each node allowing the instance on that node to use a local disk for its temporary needs. Creating multiple temporary tablespaces is of value if you want to split your user community across the available tablespaces for some reason. However, if the individual temporary tablespaces can only be as large as a single one would be, you may want to combine them into one. This will help prevent large operations from consuming all of the temporary space available to the user.



Summary

In this lesson, you should have learned how to:• Select options for populating a data warehouse• Configure segments for a data warehouse

– Denormalized data– Materialized views and summary tables– Indexes– Partitioned segments– Temporary segments

• Implement parallel processing– Configure automatic parallelism – Set manual configuration parameters


Monitoring and Tuning



Objectives

After completing this lesson, you should be able to do the following:• Create Real Application Clusters views• Select appropriate monitoring and tuning tools

– Oracle Performance Manager– Statspack– UTLBSTAT/UTLESTAT– Direct queries against dynamic performance tables

• Monitor key performance statistics• Identify and resolve key contention issues



Create Real Application Clusters Views

$ more $ORACLE_HOME/rdbms/admin/catclust.sqlRem . . .Rem NAMERem catclust.sql - CLUSTer database specific Rem views definitionsRemRem DESCRIPTIONRem Create all cluster database specific viewsRemRem NOTESRem This script must be run while connected as Rem SYS.

Create Real Application Clusters ViewsThere are a number of views that can help monitor activity specific to a Real Application Clusters database and its instances. These are created with the script catclust.sql, which you can find in the same directory as the other data dictionary and related scripts. For example, on UNIX, the path and file name would be $ORACLE_HOME/rdbms/admin/catclust.sql. To create the views, execute the catclust.sql script from a session connected as the sys user.When examining performance in a Real Application Clusters database, you have the option of using either the normal dynamic performance tables (V$ views) or global views (GV$ views). The output of the latter includes rows from each available instance. The GV$ views contain an additional column, compared to the corresponding V$ view, to indicate the instance to which each row applies. As mentioned in an earlier lesson, the GV$ views use parallel process servers, so you should include a non-zero value for the PARALLEL_MIN_SERVERS initialization parameter if you intend to query these views.



Oracle Performance Manager

• Part of the Diagnostics Pack• Optionally runs with Oracle Enterprise Manager• Graphically displays instance (V$) or cluster

database-wide (GV$) statistics

Oracle Performance ManagerThe Oracle Performance Manager, part of the Diagnostics Pack, is an available option to Oracle Enterprise Manager. Oracle Performance Manager can be run with or without Oracle Enterprise Manager. If you choose to run it as a standalone product, Oracle Enterprise Manager does not have to be configured.You can use Oracle Performance Manager to query the dynamic performance (V$) views of an individual instance or the global dynamic performance (GV$) views to retrieve the related V$ view information from all instances.Oracle Performance Manager displays the retrieved information in a variety of tabular and graphic performance charts for Real Application Clusters. The statistics represent the aggregate performance of all instances of a cluster database running on cluster. The statistics are displayed in individual charts and include information about data block pings, lock activity, file I/O, and session and user information. You can also use the Performance Manager to display an overview of several key statistics on one chart.There are approximately thirty separate charts available for you to display cluster-related statistics. These are described in detail in Chapter 7 of the Oracle9i Real Application Clusters Deployment and Performance manual.



Activate Oracle Performance Manager

• Start Oracle Performance Manager • In the navigator, expand Databases or Instance to

locate Real Application Clusters folder• In the Oracle Clusters object folder, select a chart• Click Show Chart to see chart displayed in a

separate window

Activate Oracle Performance ManagerTo use the Oracle Enterprise Manager Console, start the following components:

• Oracle Intelligent Agent• Oracle Performance Manager

To display Oracle Performance Manager charts:1. Start Oracle Performance Manager from the console or in standalone mode. 2. In the navigator, expand Databases or Clusters Instance > Cluster Database. Whether

you see the Database of the Instance folder will depend on how you started Oracle Performance Manager.

3. In the Oracle Clusters object folder, select a chart, then click Show Chart.4. The chart is displayed in a separate window.



Statspack

• Oracle utility to gather database statistics– Collects values at start and end of monitoring period– Prepares report on activity during this period– Maintains information in database tables

• spdoc.txt supplements the published documentation

• Create a Statspack environment with the spcreate.sql script

• To run Statspack for an instance– Execute Statspack packages from a SQL*Plus session– Use DBMS_JOB or operating system scheduler to

gather statistics

StatspackThe Statspack utility is a tool provided by Oracle to help you manage your database instances by collecting and analyzing statistics gathered while the instance is active. The current values from various data dictionary and dynamic performance tables are saved in special tables at the start of the analysis period and again at the end of the analysis period. The differences between the start and end values represent the work during the analysis period and Statspack uses this information to report on the characteristics of the instance during that time.In a Real Application Clusters database, the utility only collects and analyzes information for individual instances, but much of the information can be used to examine overall cluster performance. You should use Statspack to collect baseline statistics from each instance during periods of normal load and monitor the results over time to ensure that there are no unexplained changes. You can also use Statspack to monitor the system when you are experiencing degraded performance because it does not add any ongoing overhead to the system during the data collection period. However, to be able to perform meaningful comparisons between baselines and other types of activity, you need to make the collection periods identical. This allows you to convert the raw results into normalized values, that is, activity over a unit of time (second or minute) or work done by a transaction.



Configure Statspack

SQL> @spcreate.sql. . .... Creating PERFSTAT user. . .Grant succeeded.

Below are the list of online tablespaces in this database. Decide which tablespace you wish to create the STATSPACK tables and indexes. This will also be the PERFSTAT user's default tablespace.. . .Specify PERFSTAT user's default tablespaceEnter value for default_tablespace:

Configure StatspackThe directory where you find your other Oracle-supplied scripts contains a documentation file for Statspack, spdoc.txt, as well as the spcreate.sql script that creates all of the components needed to run Statspack. These include a new user, perfstat, as well as the packages to manage the data collection and analysis routines. You only need to execute the spcreate.sql script on one of your instances. The spcreate.sql script will request the names of the default and temporary tablespaces for the perfstat user. The former will require about 64 MB of free space to install the basic segments for Statspack and more for each set of statistics you collect. If you don’t have this amount of space available, you will need to add it to an existing data file or create a new tablespace. Remember that the associated files will have to be on a shared file system, which may require the use of a raw partitions of the appropriate size.



Collect Statspack Statistics

SQL> CONNECT perfstat/perfstatConnected.SQL> VARIABLE snap_id NUMBERSQL> BEGIN :snap_id := statspack.snap; END;

2 /

PL/SQL procedure successfully completed.

SQL> PRINT snap_id

SNAP_ID----------

21

Collect Statspack StatisticsYou can run Statspack from a SQL*Plus session directly by executing the required packages, or through a scheduling system, such as Oracle's DBMS_JOB package or an operating system tool.To gather statistics for an instance, connect as the perfstat user and execute the statspack.snap routine:

execute statspack.snap

You can also run statspack.snap as a function, as shown in the example, in order to retrieve the specific ID number for the set of collected data. After the desired data-collection period, re-run the statspack.snap routine to collect the end values for the statistics.The spauto.sql script initiates an hourly collection of statistics for Statspack. To use this script, set the initialization parameter JOB_QUEUE_PROCESSES to a nonzero value and run the script as the user perfstat. Note: For the most useful results, your instance should be running with TIMED_STATISTICS=TRUE in the initialization parameter file.



Generate Statspack Report

SQL> @spreport.sql

. . .DB Snap Snap

Instance Name Id Snap Started Level-------- ---- ---- ----------------- -----P1N2 P1 1 25 Nov 2001 10:00 5

2 25 Nov 2001 11:00 5

Specify the Begin and End Snapshot Ids~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Enter value for begin_snap: 1. . .Enter value for end_snap: 2. . .

Generate Statspack ReportTo generate a report based on statistics collected between two snapshots, you need to execute the spreport.sql script as the perfstat user. The following SQL*Plus session will initiate a report on a UNIX system:

SQL> CONNECT perfstat/perfstatConnected.SQL> @$ORACLE_HOME/rdbms/admin/spreport.sql

The script will list the statistical snapshots created by the statspack.snap routine on the connected instance. It will then prompt you to select the ones that bracket the time period for which you want the report, that is, the one containing the start values and the one containing the end values. The report will be based on the statistics accumulated between these two snapshots. You will also be prompted for the name of the file where you want the report to be stored.Note: If you start instances with different settings for the INSTANCE_NUMBERinitialization parameter, the snapshot numbers displayed may not be for the current instance and may even represent snapshots collected by different instances. In such cases, you cannot rely on the report to provide you with meaningful results.



UTLBSTAT and UTLESTAT

• Gathers detailed statistics over a given time interval from many v$ tables

• Start collection with utlbstat.sql• Stop collection with utlestat.sql• These utilities cannot collect concurrent statistics

on Real Application Clusters instances

UTLBSTAT and UTLESTATAnother statistics collecting and reporting system is available through the scripts utlbstat.sql and utlestat.sql. These scripts are available in the same directory as the Statspack routines. The utlbstat.sql script collects the initial set of statistics and the utlestat.sql script collects the final set of statistics and creates the report file. As with Statspack, you should run the two scripts at the start and end of a fixed monitoring period in order to establish a baseline that you can use to compare with statistics collected over the same period at a later time. Unlike Statspack, these routines only use one set of tables for the initial and final sets of statistics. This means that you cannot keep statistics in the database for long-term comparisons. Additionally, these tables can only be populated by one instance at a time. Consequently, you cannot collect statistics from two or more instances concurrently. Oracle therefore recommends that you employ Statspack in place of the utlbstat/utlestat utility for your Real Application Clusters databases, and for other databases as well.



Monitor Consistent Block Processing

SQL> SELECT NAME,VALUE FROM V$SYSSTAT 2 WHERE NAME LIKE '%global cache%';

NAME VALUE------------------ ----------global cache gets 6709global cache get time 1023global cache converts 118global cache convert time 67global cache cr blocks received 2681global cache cr block receive time 543global cache current blocks received 442global cache current block receive time 555global cache cr blocks served 620global cache cr block build time 0

Monitor Consistent Block ProcessingRequests for global resources for data blocks originate in the buffer cache of the requesting instance. Before a request enters the Global Cache Service request queue, Oracle allocates data structures in the System Global Area (SGA) to track the state of the request and collects statistics on these resource structures. To monitor global cache statistics, query the V$SYSSTAT view and analyze its output as follows:

1. Query V$SYSSTAT with a statement such asSELECT NAME,VALUE FROM V$SYSSTAT WHERE NAME LIKE '%global cache%';

The output will be similar to that shown in the example on the slide.2. Calculate the average amount of time (latency) taken to fulfill a consistent block

request. Use the following expression to compute this value in milliseconds:(global cache cr block receive time * 10) ÷ (global cache cr blocks received)

Using the values from the output above, the average time for consistent block requests in the instance is 5430 ÷ 2681 = 2.03 milliseconds.

3. Determine the cause of high average latency values, as discussed on the following pages.



Global Cache Service Request Latency

• Formula to compute the average latency to fulfill a consistent block request

(global cache cr block receive time) / (global cache cr blocks received)

• The result is measured in hundredths of a second; multiply by 10 to convert to milliseconds

Global Cache Service Request LatencyThe latency of a consistent block request (between the time of the original request and the receipt of the consistent block image at that instance) should typically be about 15 milliseconds, although this depends on your system configuration and volume. For example, if your CPU has limited idle time and your system typically processes long-running queries, the latency may be higher. On the other hand, it is possible to have an average latency of less than one millisecond with user-mode IPC.If your request latency value is too high, you may have a high value for the DB_FILE_MULTIBLOCK_READ_COUNT parameter. This is because a requesting process can issue more than one request for a block depending on the setting of this parameter, causing the requesting process to wait longer.High request latency may also be caused by a high number of incoming requests or multiple nodes dispatching requests to the LMS process for Global Cache Services. The formulae on the following page can help you determine if the delay is caused by LMS overhead, including the network time.Note: In OLTP systems with multiblock locking and a high degree of contention, it is not unusual for global cache wait events to represent a high proportion of the total time waited.



Global Cache Service Request Latency

• Average time to build consistent read blockA = global cache cr block build time / global cache cr blocks served

• Average time to wait for log flushB = global cache cr block flush time / global cache cr blocks served

• Average time to send completed blockC = global cache cr block send time / global cache cr blocks served

• Average LMS service timeaverage latency – A – B – C

Global Cache Service Request Latency (continued)Compute the average service time for an LMS request using the following calculations:

• Compute the average time to build a consistent read block (first formula above)• Compute the average time waited for a log flush (second formula above)• Compute the average time to send a completed block (third formula above)

The difference between the average latency time and the sum of the above three average times represents the time spent by the LMS service and the network activity. The former may be caused by an overburdened CPU and the latter by a slow or overtaxed cluster interconnect.



Monitor Current Block Processing

• Current block average latency( global cache current block receive time / global cache current blocks received ) – ( ( global cache current block pin time + global cache current block flush time + global cache current block send time ) / global cache current blocks served )

• High values indicate possible contention for buffers, CPU cycles, or interconnect access

Monitor Current Block ProcessingYou can compute the latency involved in processing the requests for current blocks in a similar manner to that used to compute read consistent block request latency. The formula above computes the overall average latency and subtracts from it the nonservice specific components.If you discover high latency values, they may indicate potential contention, typically for the buffer cache, for the CPU, or for the interconnect.



Monitor Block Mode Conversions

• Average convert time, in milliseconds10 * (global cache convert time) / (global cache converts)

• Average get time, in milliseconds10 * (global cache get time) / (global cache gets)

• Timeouts are counted by the statistic global cache convert timeouts

Monitor Block Mode ConversionsYou can calculate the average convert time for a block mode conversion using the first formula shown above. The value should be between ten and 20 milliseconds. High values indicate either overall system workload or performance problems or may be caused by excessive use of the Global Cache Service or the same block resources by multiple instances.The second formula above calculates the average the time taken to retrieve a block using Global Cache Services. Ideally, this value should be between 20 and 30 milliseconds. Larger values indicate excessive contention for Global Cache Service operations by multiple instances.The statistic global cache convert timeouts records the number of times that block mode conversion operations have timed out. This value should be zero and any nonzero value indicates serious performance problems with the system as a whole. This could be caused by hardware or network failures, by excessive workloads on the system on the interconnect, or by high contention for blocks between instances.



Analyze Global Enqueue Statistics

• Collect global statistical data by querying V$SYSSTAT

• Calculate average global enqueue time• Calculate the average global lock convert time• Identify the types of resources that may be causing

problems• Perform further analysis with the V$LIBRARYCACHE

and V$ROWCACHE views

Analyze Global Enqueue StatisticsThe Global Enqueue Service (GES) manages all the non-Cache Fusion intra- and inter-instance resource operations. High GES workload request rates can adversely affect performance.Global resource statistics include global enqueue requests originating from all layers of the kernel while global cache statistics relate to buffer cache activity. The following steps allow you to monitor data from the V$SYSSTAT view to derive Global Resource Service latencies, counts, and averages.

1. Collect global statistical data by querying V$SYSSTAT2. Calculate average global enqueue time3. Calculate the average global lock convert time4. Identify the types of resources that may be causing problems5. Perform further analysis with the V$LIBRARYCACHE and V$ROWCACHE views

These statistics can help you estimate the workload generated by a Real Application Clusters instance.



Collect Global Enqueue Statistics

SQL> SELECT name, value FROM v$sysstat 2 WHERE name LIKE '%global lock%';

NAME VALUE---------------------------------- ----------global lock sync gets 17745global lock async gets 2677global lock get time 2092global lock sync converts 2391global lock async converts 1367global lock convert time 346global lock releases 17291

7 rows selected.

Collect Global Enqueue StatisticsUse this syntax to query V$SYSSTAT to find the statistics related to global enqueues:

SELECT name, value FROM v$sysstat WHERE name LIKE '%global lock%';

The output is similar to that shown in the slide.



Calculate Average Global Enqueue Time

Average global enqueue time, in milliseconds10 * (global lock get time) / (global lock sync gets +

global lock async gets)

Calculate Average Global Enqueue TimeCompute the average global enqueue get time using the formula shown on the slide, where the values are provided from the output to the query against V$SYSSTAT.Using the data from our previous query against V$SYSSTAT, the result of this calculation is

10 * 2092 / (17745 + 2677) = 1.02 milliseconds

If the result of the calculation, which includes a multiplier of ten to convert it to milliseconds, is greater than 20 or 30 milliseconds, then you need to try and identify the cause, as discussed later.



Calculate Average Global Lock Convert Time

Average global lock convert time, in milliseconds10 * (global lock convert time) / (global lock sync

converts + global lock async converts)

Calculate Average Global Lock Convert Time Calculate average global lock convert time using the formula shown on the slide. Multiply the result by 10 to convert the result to milliseconds.Using the data from our previous query against V$SYSSTAT, the result of this calculation is

10 * 346 / (2391 + 1367) = 0.92 milliseconds

The result should be less than 20 milliseconds. If it is much greater than this, or if it increases over time, you should try to determine the cause as discussed on the next page.



Calculate Average Global Lock Convert Time

SQL> SELECT event, time_waited, average_wait2 FROM v$system_event3 ORDER BY time_waited DESC

EVENT TIME_WAITED AVERAGE_WAIT--------------------- ----------- ------------rdbms ipc message 65249582 18PX Idle Wait 42068468 195ges remote message 8422524 34smon timer 8111119 28560pmon timer 6303741 0async disk IO 96758 0PX Deq: Execution Msg 88597 91direct path write 39321 1

Calculate Average Global Lock Convert Time (continued)If the average global enqueue get time is larger than 20 to 30 milliseconds or the average global lock convert time is much greater than 20 milliseconds, you should try to identify the cause as described below.First, query the TIME_WAITED column in the V$SYSTEM_EVENT view using the DESCkeyword to help identify the event causing the delay. The results of such a query are shown in the example. Events with the highest wait times are the most likely candidates for low performance in the database. You can ignore events that are not associated with the Global Enqueue Service during this tuning activity.Second, attempt to discern the types of resources that may be causing the high conversion times, either synchronous or asynchronous operations. Do this by replacing the sums in the denominators of the formulae for calculating the average global enqueue and lock conversion times. Use the value from the synchronous or from the asynchronous portion of the denominator to compute the average synchronous and asynchronous resource usage respectively. The former are usually locking events and the latter are mainly from nonblocking inter-instance process activity.



Further Analysis with V$LIBRARYCACHEand V$ROWCACHE

SQL> SELECT namespace, dlm_lock_requests, 2 dlm_pin_requests,3 dlm_pin_releases, 4 dlm_invalidation_requests,5 dlm_invalidations6 FROM v$librarycache;

SQL> SELECT parameter, dlm_requests, 2 dlm_conflicts, dlm_releases3 FROM v$rowcache

Further Analysis with V$LIBRARYCACHE and V$ROWCACHEAnalyze the V$LIBRARYCACHE and V$ROWCACHE views to observe Global Enqueue Service activity on local resources. These views have Global Enqueue Service-specific columns, with column names that beginning with the string DLM_, that identify Global Enqueue Service resource use. Analyze these views for Global Enqueue Service activity if you have frequent and extended waits for library cache pins, enqueues, or pointers to global resources (reported as DFS lock handles).



Monitor Global Enqueue Service Resource Statistics

• Enable collection of Global Enqueue Service statistics

EVENT="29700 TRACE NAME CONTEXT FOREVER"

• Query statistical results– V$GES_CONVERT_LOCAL– V$GES_CONVERT_REMOTE

Monitor Global Enqueue Service Resource StatisticsTwo views, V$GES_CONVERT_LOCAL and V$GES_CONVERT_REMOTE, provide useful statistics for monitoring Global Enqueue Service latency and workload. The V$GES_CONVERT_LOCAL view shows statistics for local Global Enqueue Service enqueue operations, that is, operations that are performed entirely within the local node without sending messages. The V$GES_CONVERT_REMOTE view shows values for remote Global Enqueue Service enqueue conversions that require sending messages to and waiting for responses from other nodes.. As with other views examined so far, they provide timing information in 100ths of a second.In order to populate the V$GES_CONVERT_LOCAL and V$GES_CONVERT_REMOTEviews, you must enable event 29700. Do this by adding the following entry to yourinitialization parameter file:EVENT="29700 TRACE NAME CONTEXT FOREVER"



Analyze Global Enqueue Service Resource Statistics

SELECTr.convert_type,r.average_convert_time,l.average_convert_time,r.convert_count,l.convert_count,

FROM v$ges_convert_local l, v$ges_convert_remote r

WHERE r.convert_count <> 0 OR l.convert_count <> 0

GROUP BY r.convert_type;

Analyze Global Enqueue Service Resource StatisticsCalculate the ratio of local-to-remote global enqueue resource operations using the query shown in the example. Save the results the first time you execute this query as a baseline. In fact, it is useful to maintain a history of these ratios in order to monitor for changes workload counts and processing times for block mode conversions.If conversion count rates increase but average times stay the same, it is likely that the application workload has changed in some way. If the values remain at the new level for a series of comparable samples, you should use them for a new baseline measure.A decrease of both conversion counts and times usually indicates an increased rate of resource conflicts or an increased workload due to message latencies, system problems, or timeouts. Note: If you query V$GES_CONVERT_LOCAL or V$GES_CONVERT_REMOTE directly, you will notice conversions that include the SSX mode as the “to” or “from” value. These count and time values in these rows will always be zero unless you are using an earlier version of Oracle.



Additional Views to Examine Resource Activity

V$LOCK_ACTIVITY

• Summarizes blocks up- or down-converted• X-to-N or X-to-S counts indicate attempts to change

blocks being held in exclusive mode by another instance

V$CLASS_CACHE_TRANSFER

• Summarizes conversion activity by class of block: data, segment header, extent header, undo

V$CACHE_TRANSFER

• Helps identify blocks and objects used by multiple instances

V$LOCK_ACTIVITY, V$CLASS_CACHE_TRANSFER, and V$CACHE_TRANSFER These three views provide different levels of detail. You can monitor the V$LOCK_ACTIVITY view to generate an overall Real Application Clusters workload profile by viewing and recording the rate at which block mode conversions occur.For more details, use the V$CLASS_CACHE_TRANSFER view to identify the type of block on which block mode conversions are occurring. Once you have identified the class, use the V$CACHE_TRANSFER view to obtain details about a particular table or index and the file and block numbers on which there is significant block mode conversion activity.If your response time or throughput requirements are no longer being met, then examine the V$LOCK_ACTIVITY, V$CLASS_CACHE_TRANSFER, V$CACHE, V$CACHE_TRANSFER or V$FILE_CACHE_TRANSFER views. In addition, you might also examine:•V$SYSSTAT to identify an increase in resource requests per transaction•V$SYSSTEM_EVENT to identify longer waits for global cache resources or consistent

read server requests per transaction• Global and local work done as described earlier to see if there is a noticeable change in

performance percentages.



The V$SYSTEM_EVENT View

• Transaction response timeresponse time / number of transactions = CPU time / number of transactions + wait time / number of transactions

• Total wait time and subcomponentstotal wait time / number of transactions = (db file sequential read tw) / number of transactions + (global cache cr request tw) / number of transactions + …

The V$SYSTEM_EVENT ViewData about Cache Fusion and Real Application Clusters events appears in the V$SYSTEM_EVENT view. By generating a list of wait events sorted in descending order by the TIME_WAITED column, you can easily locate performance bottlenecks. For long-term comparisons, you should divide the sum of values from the TOTAL_WAITS and TIME_WAITED columns by the number of transactions. Transactions in this sense can be defined as business transactions, for example, insurance quotes, order entry, and so on. Or you can define them on the basis of user commits or executions, depending on your perspective.The goal is to estimate which event type contributes the most to transaction response times, based on the first equation shown above. Also, the total wait time can be divided into subcomponents of the wait time as shown in the second equation shown above, where tw is time waited: It is also useful to derive the total wait time by adding the individual events and observing the percentages that are spent waiting for each event. This enables you to derive the major cost factors for transaction response times. Reducing the time for the largest proportion of the waits has the most significant effect on response time.



Real Application Clusters Events in V$SYSTEM_EVENT

• global cache cr request• library cache pin• buffer busy due to global cache• global cache busy• global cache open x• global cache open s• global cache null to x• global cache s to x• global cache null to s

Real Application Clusters Events in V$SYSTEM_EVENTThe following events appearing in the V$SYSTEM_EVENT output represent waits for Real Application Clusters events:•global cache cr request•library cache pin•buffer busy due to global cache•global cache busy•global cache open x•global cache open s•global cache null to x•global cache s to x•global cache null to s

You should also monitor these additional events because performance problems can be related to server coordination processing within Real Application Clusters. These events are:•Row cache locks•Enqueues•Library cache pins•DFS lock handle



General Observations for Tuning Inter-Instance Performance

SELECT parameter, count, dlm_requests,dlm_conflicts, dlm_releases

FROM v$rowcache

SELECT namespace, dlm_lock_requests, dlm_pin_requests, dlm_pin_releases,dlm_invalidation_requests, dlm_invalidations

FROM v$librarycache

General Observations for Tuning Inter-Instance PerformanceIf the time waited for global cache events is high relative to other types of waits, then look for increased latencies, contention, or excessive system workloads. Do this by examining V$SYSSTAT statistics and operating system performance monitors. Excessive counts for global cache busy or buffer busy waits indicate buffer cache contention.If excessive wait time is used by waits for nonbuffer cache resources as shown by statistics in the rows row cache lock, enqueues, and library cache pin, then monitor the DLM_ columns in V$ROWCACHE and V$LIBRARYCACHE views for Real Application Cluster-related issues as shown above. This output shows a few lines from the second query:

DLM_ DLM_INVALLOCK_ DLM_PIN_ DLM_PIN_ IDATION DLM_INVAL

NAMESPACE REQUESTS REQUESTS RELEASES _REQUESTS IDATIONS--------------- -------- -------- -------- --------- ---------SQL AREA 0 0 0 0 0TABLE/PROCEDURE 2921 1348 177 422 0

Real Application Clusters problems often arise from poorly managed space parameters or sequence numbers that are not cached. In such cases, processes wait for row cache locks and enqueues while the V$ROWCACHE view shows conflicts for certain dictionary caches.



Tuning Strategy

• Collect baseline data• Monitor statistics and compare to baseline data• Tune only if performance falls below acceptable

levels• Remember to monitor performance factors

unrelated to cluster database management• Address causes for high levels of contention

Tuning StrategyUse a baseline set of statistics, collected by a method of your choice, to compare performance over the life of the database or during times of reduced performance. If performance starts to fall below acceptable levels, use these statistics to determine the primary causes. These will usually be related to an increased amount of contention for a resource or set of resources. In a Real Application Clusters database, this contention may be caused by factors within one or more single instances and be completely unrelated to the components required to support a cluster database: you should address these problems before attempting to perform database-wide tuning. If you find that the contention is related to inter-instance processing, try to resolve it by the simplest methods possible, such as changing related initialization parameters in one or more instances, reorganizing disk files to reduce I/O contention, and employing features designed to help manage Real Application Clusters databases like automatic segment space management, automated undo management, and automatic parallelism. If this fails to resolve the performance problems, you should determine if the application can be improved or even partitioned across the instances to reduce the volume of inter-instance communication. In extreme cases, you may need to repartition your data and configure multiblock locks to reduce Global Resource Service activity.



Summary

In this lesson, you should have learned how to:• Create the views specific to Real Application

Clusters• Collect and view performance statistics with various

tools– Oracle Performance Manager– Statspack– UTLBSTAT/UTLESTAT– Dynamic performance tables

• Identify causes of contention and performance degradation

• Apply a tuning strategy


Case Study(Optional)



Objectives

After completing this lesson, you should be able to do the following:• Incorporate information learned during this course

in a database design project• Design a (theoretical) Real Application Clusters

database



Case Study

Develop a design for a database with the following characteristics:• Data warehouse for use by multiple analysts

performing extensive OLAP and data mining operations: some multi-day queries expected

• Extensive daily uploads from multiple OLTP sources (anticipate loads taking multiple hours)

• Hardware consists of a cluster with four symmetric (identical) nodes

MethodYour instructor will assign you to a team which will work on the problem described above. After a given amount of time, to be determined by your instructor, your group will be expected to present and justify your design decisions to the rest of the class. Your instructor will explain the type of presentation you should give and provide any materials necessary for this exercise.

Key considerations• How will you provide resources for the load activity with minimal degradation of

continuing queries?• How will you assign users to instances when they log on?• How will you assign users to a temporary tablespace when you add them?• Do you need to configure instance groups or manage parallel operations in some other

way?• Can your design handle node or instance failures with minimal impact on users

executing queries? And how will a failure impact your load operations?• Does your design support a growth path that may require adding new nodes and

instances?



Case Study

• Present your database design to the class– Explain the key features of your design– Be prepared to defend your decisions

• Compare and contrast the different design strategies presented– Is there an overall preference?– Can elements from different designs be combined?

DiscussionAt the end of the presentations, if there is time, discuss the various designs and determine if there are key elements common to all of them or if there are elements from each design that could be combined into a “class” solution.Remember, this is just an exercise to let you review what you have learned and not a real database design. There are no right or wrong answers.



Summary

In this lesson, you should have learned how to:• Select from available options to design a Real

Application Clusters database• Refine a database design based on additional input

12837GC10 - Oracle9i：Real Application Clusters

Documents

Transcript of 12837GC10 - Oracle9i：Real Application Clusters