Enabling a Highly Scalable and Available Storage ... · Storage Environment Using Oracle Automatic...

www.dell.com/powersolutions POWER SOLUTIONS 37

SCALABLE ENTERPRISE

Enabling a Highly Scalable and Available Storage Environment Using Oracle

Automatic Storage ManagementAs databases grow larger, requiring more disk drives and cluster nodes, traditionalstorage management techniques can become less efficient, more complex, and proneto human error. Automatic Storage Management (ASM), a volume manager for Oracle�

Database 10g, helps simplify and automate many of the difficult storage managementtasks surrounding large Oracle databases. This article introduces the concept of ASMand presents techniques for implementing Oracle Database 10g databases on Dell/EMCFibre Channel and Dell™ PowerVault™ SCSI storage arrays.

As database storage requirements approach thousands

of disk drives, or tens of nodes in a cluster, tradi-

tional techniques for file management can stop working.

Traditional alternatives for storing database files—such

as Oracle® Cluster File System, third-party volume man-

agers, and raw devices—do not always scale efficiently.

In addition, such approaches require independent effort

on every node within a cluster and thus become too

prone to human error. With so many disks, tasks such

as storage provisioning, disk initialization, file system

expansion, and manual load balancing can become com-

plex and cost-prohibitive.

Database administrators (DBAs) need tools that

can help increase their productivity and automate many

manual tasks. To address these needs, Oracle has intro-

duced Automatic Storage Management (ASM), a new

feature in Oracle Database 10g that helps simplify the

routine management of physical storage media. This

article describes how ASM can solve many of the prac-

tical file management problems inherent in large Oracle

databases and how ASM functionality can be applied

to Dell/EMC Fibre Channel and Dell™ PowerVault™ SCSI

storage arrays.

Introducing Oracle Automatic Storage Management ASM is an Oracle database file system and volume man-

ager that is built into the Oracle Database 10g kernel. It

provides both a graphical user interface (GUI) through

the Oracle Enterprise Manager 10g Grid Control man-

agement application1 and a simple SQL-based interface

that allows DBAs to use familiar commands such as

CREATE, ALTER, and DROP.

BY ZAFAR MAHMOOD, JOEL BORELLIS, MAHMOUD AHMADIAN, AND PAUL RAD

1 For more information, see “Scaling out with Oracle RAC 10g on Dell Clusters” by Paul Rad; Zafar Mahmood; Ibrahim Fashho; Reza Rooholamini, Ph.D.; and Joel Borellis in Dell Power Solutions, June 2004.

037-43 MahmoodBorellis.qxd 6/16/04 2:25 PM Page 37

ASM enables storage management of all Oracle files—including

data files, log files, control files, archive logs, and Recovery

Manager backup sets—across thousands of disk drives with mini-

mal management overhead. ASM also provides storage management

for multiple nodes in an Oracle Real Application Clusters (RAC) data-

base environment, as well as for single-node symmetric multi-

processing (SMP) servers.

Using ASM, DBAs can dynamically scale out their Oracle data-

base storage without affecting database availability. ASM also auto-

matically load balances I/O across all available disks defined in the

storage environment, helping to prevent hotspots and to maximize

performance despite rapidly changing data-usage patterns. Addi-

tionally, ASM helps reduce disk fragmentation such that data relo-

cation is unnecessary for reclaiming unused space. ASM facilitates

automatic online disk space reorganization for the incremental addi-

tion or removal of storage capacity.

DBAs can use ASM to maintain redundant copies of data for

increased fault tolerance at the database level, or they can config-

ure ASM on top of hardware-based RAID technologies such as those

available in Dell/EMC and Dell PowerVault storage arrays (see

Figure 1). Moreover, ASM automatically tunes I/O operations based

on performance characteristics for specific classes of data—such as

redo, data, control, archive, and temporary data files—without

requiring human interaction on a per-file basis.

Enabling an ideal database storage environmentASM helps address the critical requirements of an optimal database

storage environment, which should provide the following features:

• Dynamic provisioning: The storage environment should

dynamically scale the storage without affecting database

availability.

• I/O balancing: The storage environment should load balance

the I/O across available I/O channels and spindles using an

optimal striping methodology, such as hardware RAID imple-

mented at the storage-array level and EMC® PowerPath®

path-management software configured to load balance the

I/O across host bus adapters (HBAs). If the number of I/O

channels or disk spindles increases, the storage environment

should be able to load balance the I/O across the new chan-

nels and spindles as needed with minimal intervention.

• High availability: The storage environment should provide

fault tolerance and high availability in case of disk or I/O

channel failures using optimal mirroring techniques, such as

hardware RAID implemented at the storage-array level and

EMC PowerPath software configured to load balance the I/O

across HBAs.

• High performance—application I/O: The storage environ-

ment should be optimized according to application I/O

behavior and needs. For example, Oracle RAC 10g databases

require direct I/O that bypasses the file-system buffer cache.

• High performance—direct I/O: Current mechanisms to pro-

vide direct I/O support for Oracle database files are either too

complex, as with raw devices, or not well integrated with

Oracle database releases. An ideal storage mechanism should

be able to provide high direct-I/O performance without com-

promising manageability or robustness.

• High performance—asynchronous I/O: The storage envi-

ronment should be able to perform kernel mode asynchro-

nous I/O. A file system–based storage environment requires

special configuration files, device drivers, or kernel parameter

tweaks to enable asynchronous I/O—adding further complex-

ity to the existing storage environment.

Dynamic provisioning and storage scalability ASM virtualizes database storage into disk groups, which serve as

repositories for Oracle database files. A disk group is a logical group-

ing of several individual disks in a storage array or of several RAID

logical storage units (LUNs). ASM enables DBAs to manage a small

set of disk groups, and ASM automates the placement of the data-

base files within those disk groups.

Adding or removing disks or LUNs to and from a disk group

is a dynamic process that does not affect database availability.

Figure 2 shows an example in which storage is added to an existing

disk group, groupA, on the fly. Without shutting down the database,

SCALABLE ENTERPRISE

POWER SOLUTIONS June 200438

Figure 1. Basic architecture of ASM

Raw device or ASMlibrary driver interface



ASM disk groups

ASM instance

Oracle Database 10g using ASM for database file storage

Oracle Enterprise Manager to manage ASM instance and disk groups

/dev/sdb /dev/sdc /dev/sdd

Linux database server running Oracle Database 10g using ASM

Storage processor 2 Storage processor 1

Dell/EMC storage array

Logicallayer

LUN LUN LUN

Physicallayer

Disk Disk Disk Disk Disk Disk Disk Disk Disk Disk

Storage array processor ports


ASM automatically formats the new disks and rebalances the disk

group by striping data across all available disks.

High storage availabilityDisk groups have three redundancy options for fault tolerance and

availability.

External redundancy. When ASM incorporates external storage

LUNs that already provide fault tolerance using RAID techniques, DBAs

can configure disk groups to use the external redundancy of the stor-

age array. DBAs should use external redundancy when creating disk

groups on Dell/EMC Fibre Channel storage arrays or Dell PowerVault

SCSI storage arrays, which have hardware-based RAID provided by

Dell PowerEdge Expandable RAID Controller (PERC) cards. See

“Considerations when using external redundancy” for storage array

configurations that can help achieve high performance and data

availability when using external redundancy.

Figure 3 shows a command that creates a disk group, groupA,

with three RAID LUNs that are configured on a Dell/EMC Fibre

Channel storage array.

Normal redundancy. When the external storage array does not

use RAID, DBAs can configure disk groups as a two-way mirror.

Drives in a disk group can be divided into failure groups, which

are disk collections that can become unavailable through failure of

an associated component, such as disks or controllers. To achieve

high availability, the disks of each failure group within a given disk

group should reside on separate controllers.

In an environment with multiple storage arrays, DBAs can imple-

ment failure groups by creating a single disk group across all arrays.

A failure group within the disk group corresponds to disks from just

one array. This usage protects against failure of a storage array.

The sample command shown in Figure 4 uses two disk con-

trollers: controller1 and controller2. Each controller is connected

SCALABLE ENTERPRISE


ASM Command:ALTER DISKGROUP groupA ADD DISK '/dev/raw/raw4', '/dev/raw/raw5';

Disk formatting

Disk group rebalancing

Figure 2. Adding storage to a disk group on the fly

CREATE DISKGROUP groupA EXTERNAL REDUNDANCY DISK'/dev/raw/raw1','/dev/raw/raw2','/dev/raw/raw3';

Figure 3. Creating a disk group with external redundancy

CREATE DISKGROUP groupA NORMAL REDUNDANCYFAILGROUP controller1 DISK

'/dev/raw/raw1','/dev/raw/raw2','/dev/raw/raw3'

FAILGROUP controller2 DISK

'/dev/raw/raw4','/dev/raw/raw5','/dev/raw/raw6';

Figure 4. Creating a disk group with normal redundancy

ASM does not use failure groups with the external redundancy option.Instead, it depends on the redundancy provided by the intelligent storagearray. Storage arrays hide the physical boundaries of disks required forproper placement of data. Therefore, for optimal performance and dataavailability, the storage arrays should be created with disks that are identi-cal in size, speed, and data-transfer capacity. For example, a RAID-0+1array of four 146 GB disks yields a capacity of 292 GB, which can be sub-divided into two logical disks: one for the database and the other for theflash recovery area. Both logical disks can then be put in the same ASMdisk group. The inner portion of this logical disk should be used as the

flash recovery area, and the outer portion, which exhibits better perfor-mance, should be used as the database area.

A logical disk that resides on the outer portion of the drives in thearray performs better than one located elsewhere on the drive becausethe outer portion of a drive has a higher rotational speed and more sec-tors per track; logical disks corresponding to inner tracks have a slowertransfer rate. Therefore, for better performance, the database areashould be the logical disk that is located on the outer portion of thearray. Logical disks with different performance characteristics shouldbe put in separate disk groups.

CONSIDERATIONS WHEN USING EXTERNAL REDUNDANCY


to an equal number and size of

disks. When the disk group is

created, the normal redundancy

option is specified along with

the FAILGROUP keyword. Each

failure group consists of an

independent disk controller and

disks that ASM uses for a two-

way mirror. When the Oracle

database creates or updates files

on a normal-redundancy disk

group, it automatically mirrors

data between both failure

groups to provide high avail-

ability if a disk or controller

fails.

High redundancy. The high-redundancy option provides

three-way mirroring for a disk group. This option requires that

three sets of disks of the same size reside on three separate con-

trollers. When the Oracle database creates or updates files on a

high-redundancy disk group, it automatically mirrors data across

three failure groups to provide a higher degree of protection than

the normal-redundancy option.

High storage performance ASM provides a built-in mechanism for implementing the Oracle

stripe and mirror everything (SAME) methodology, in which all

types of data are striped and mirrored across all available drives.

The SAME approach can result in an I/O load that is evenly dis-

tributed and balanced across all disks within the disk group.

ASM also can be used to create disk groups across multiple

storage array–based RAID LUNs. This technique, called plaiding,

results in double striping: striping at the storage-array level and

striping at the host level.

In a disk group with external redundancy, the SAME approach

should be used at the storage-array level. All LUNs created in

the Dell/EMC storage array should be configured as RAID-10. If

the storage array’s RAID is not being used, then disk groups

should be created using normal or high redundancy across mul-

tiple I/O controllers, which may result in optimal I/O performance

for Oracle Database 10g.

ASM also uses direct I/O to access Oracle database files; file

systems lack this capability without tweaking extra kernel param-

eters or relinking Oracle binaries with asynchronous I/O (AIO)

libraries. In an Oracle RAC environment, direct I/O is required to

avoid corrupting database files, because the file-system buffer cache

is local to each node and other nodes in the cluster do not have

access to it.

Moreover, ASM provides kernel asynchronous I/O by default

without requiring special parameter settings. Using asynchronous—

or nonblocking—I/O can improve performance over serial I/O:

asynchronous I/O requires fewer I/O processes to handle multiple

I/O requests because pending I/Os do not need to wait in the queue.

Simplifying storage provisioningCompared to other storage options such as raw devices, Oracle

Cluster File System, and third-party volume managers, ASM can

greatly simplify the process of scaling storage, as shown in Figures

5 and 6. ASM automates most of the manual steps required by the

SCALABLE ENTERPRISE


Before ASM

Add more LUNsto the operating system

Add more LUNsto the operating system

With ASM

Create a new volume using athird-party volume manager

Issue the ALTER DISKGROUPgroupname ADD DISK command

Create a file system overthe new volume

Decide which data should bemoved to the new volume

Move data to thenew volume

Tune the file system to rebalance the I/O

Figure 5. Scaling up storage for Oracle Database 10g

Before ASM

List data on the diskto be removed

With ASM

Configure the additionalfile-system location to holddata from the dropped disk

Remove the disk from the operatingsystem when rebalance completes

Move data to newlocation on the file system

Rename files inthe database

Remove the disk fromthe operating system

Tune the file system torebalance the I/O

Issue the ALTER DISKGROUPgroupname DROP DISK command

Figure 6. Scaling down storage for Oracle Database 10g

ASM automates most

of the manual steps

required by the other

approaches, so that DBAs

do not need to make

complex provisioning

decisions and are freed

for other critical tasks.

037-43 MahmoodBorellis.qxd 6/16/04 2:25 PM 1

SCALABLE ENTERPRISE


This sidebar describes the basic procedure for configuring and using ASMwith Oracle Database 10g databases running on the Red Hat� EnterpriseLinux� operating system. In this example, all three LUNs—/dev/sdb, /dev/sdc,and /dev/sdd—have been configured as RAID-10 according to the Oraclerecommendation for using the SAME methodology. Because RAID is pro-vided at the storage-array level, disk groups are created using the external-redundancy option.

1. Verify that the external storage devices are visible on the Linux host:

cat /proc/partitions

The configured storage LUNs on the Dell/EMC or Dell PowerVaultstorage array should appear as follows:

/dev/sdb

/dev/sdc

/dev/sdd

Partitions do not need to be created on the storage devices becauseASM can incorporate whole disks when creating or expanding a disk group.

2. ASM on Linux hosts requires that the external storage devices eitherbe bound to raw character devices or be configured using the ASMlibrary driver. To help easily identify the raw devices, rename themusing ASM-friendly names:

mv /dev/raw/raw1 /dev/raw/ASM1



3. Make user oracle of group dba the owner of the candidate disks:

chown oracle.dba /dev/raw/ASM*

4. Bind the storage devices to the raw character devices:

raw /dev/raw/ASM1 /dev/sdb

raw /dev/raw/ASM2 /dev/sdc

raw /dev/raw/ASM3 /dev/sdd

5. To make the binding consistent on each reboot, edit /etc/sysconfig/rawdevices to add these entries:

/dev/raw/ASM1 /dev/sdb

/dev/raw/ASM2 /dev/sdc

/dev/raw/ASM3 /dev/sdd

At this point, three storage devices are available for ASM.

6. Create an Oracle Database 10g database using the Oracle DatabaseConfiguration Assistant (DBCA). DBCA automatically performs thefollowing operations:

• If ASM is chosen as the database file-storage method (see Figure A), DBCA automatically creates and starts an ASM management instance (see Figure B).

• DBCA incorporates and formats the specified storage devices, creates disk groups as specified by the DBA (see Figure C), andprepares them for the Oracle Database 10g database.

• DBCA creates the database using disk groups selected by theDBA as data file storage, as shown in Figure D.

Figure A. Selecting ASM as the storage mechanism Figure B. Creating and starting an ASM instance

CREATING AN ORACLE DATABASE 10g DATABASE USING ASM


other approaches so that DBAs do

not need to make complex provi-

sioning decisions and are freed for

other critical tasks.

Providing efficient managementof a large database environmentBuilding a scalable storage envi-

ronment for databases has been

a challenge for DBAs, especially

in a RAC environment in which

multiple nodes access the data-

base. The management of database storage using existing stor-

age management techniques, which are detached from the

databases, becomes very complicated and impractical when data-

base size grows into the terabyte range.

Oracle Database 10g can support databases with sizes

ranging into multiple exabytes,2 because it can support a single

data file that contains 4×109 Oracle blocks; previously, the

limit was 4×106 Oracle blocks. To simplify the management of

these large databases, Oracle tightly integrated ASM with the

database instead of relying on storage management based on

the operating system. ASM enables the Oracle database to

automatically provision the required storage to the database

files as needed from a pool of disk groups that may be

expanded, shrunk, or rebalanced for optimal I/O on the fly—

freeing DBAs from many of the complex, routine tasks of

storage management.

Zafar Mahmood ([email protected]) is a software engineer in the Dell Database

and Application Engineering Department of the Dell Product Group. He has been involved in

database performance optimization, database systems, and database clustering solutions

for more than six years. Zafar has an M.S. in Electrical Engineering with a specialization in

Computer Communications from the City University of New York.

Joel Borellis ([email protected]) is a technology director for Oracle, working exclu-

sively with Dell for the past two years. He is in the Oracle Server Technologies Organization,

where he works on Oracle and Dell solutions in the areas of development, engineering, and

sales and marketing.

Mahmoud Ahmadian ([email protected]) is a development engineer in the

Dell Database and Application Engineering Department of the Dell Product Group. He has an

M.S. in Computer Science from the University of Houston.

Paul Rad ([email protected]) is a senior software engineer in the Dell Database and

Application Engineering Department of the Dell Product Group. Paul’s current interests are in

the area of operating systems, database systems, clustering, storage technologies, and vir-

tualization. Paul has master’s degrees in both Computer Engineering and Computer Science

from The University of Texas at San Antonio.

SCALABLE ENTERPRISE


Figure C. Selecting member disks of a disk group Figure D. Selecting disk groups for database storage

FOR MORE INFORMATION

Oracle Database 10g: http://www.oracle.com/database

Oracle Automatic Storage Management:http://otn.oracle.com/obe/obe10gdb/manage/asm/asm.htm

2 One exabyte = 260 (1,152,921,504,606,846,976) bytes, or 1,024 petabytes.

ASM enables storage

management of all

Oracle files across

thousands of disk drives

with minimal manage-

ment overhead.


Enabling a Highly Scalable and Available Storage ... · Storage Environment Using Oracle Automatic...

Documents

Transcript of Enabling a Highly Scalable and Available Storage ... · Storage Environment Using Oracle Automatic...