Using SecureFiles on a Sun Storage 6780 Array...Oracle White Paper—Using SecureFiles on a Sun...

An Oracle White Paper March 2010

Using SecureFiles on a Sun Storage 6780 Array

Oracle White Paper—Using SecureFiles on a Sun Storage 6780 Array

Introduction ......................................................................................... 1 Overview of SecureFiles ..................................................................... 2 Overview of SecureFiles Architecture ................................................. 2 Hardware Configuration ...................................................................... 5 I/O Limits ............................................................................................. 5 Performance Evaluation...................................................................... 6

Single-Node Performance............................................................... 7 Multiple-Node Performance .......................................................... 11

Performance Guidelines.................................................................... 13 General Recommendations .......................................................... 13 Recommendations Based on SecureFiles Size ............................ 14

Conclusion ........................................................................................ 15 Appendix A: Hardware Setup............................................................ 16 Appendix B: Oracle Setup and Test Description............................... 18 Appendix C: White Papers ................................................................ 20


1

Introduction

The Oracle Database 11g SecureFiles feature provides customers with superior performance, scalability, high availability, and control over structured and unstructured content. Using Oracle Database 11g for file system storage and Oracle’s Sun Storage 6780 array provides performance that is far greater than conventional file systems. This approach delivers superior scalability, optimizes the use of the database for storage of all enterprise data (structured and unstructured), and offers other advanced database capabilities not possible with traditional file systems.

This white paper presents new levels of ingestion and retrieval rates on a single node and scalability across multiple nodes of SecureFiles. The document includes the following information:

• An overview of the SecureFiles architecture

• Highlights of various aspects of hardware setup and storage configuration

• A description of benchmark variables and tests conducted

• A summary of single- and multiple-node results

• A description of performance guidelines based on benchmark experiments


2

Overview of SecureFiles

SecureFiles, a key feature of Oracle Database 11g, lets you store unstructured file content such as images, audio, video, PDFs, and spreadsheets. SecureFiles delivers unparalleled performance for file data compared to traditional file systems, while retaining the reliability, scalability, and security advantages of Oracle Database 11g. In addition, SecureFiles provides a rich set of new features including compression, encryption, and deduplication. SecureFiles supersedes the large object (LOB) datatype introduced in Oracle 8. Given the rapid growth of Web services and digital media, coupled with the explosion in data volumes, applications today demand hardware configurations that can support 2 Gb/sec to 8 Gb/sec throughputs. SecureFiles is well suited to meet this demand as demonstrated by the breakthrough performance result of 4.5 Gb/sec on the host.

To achieve the performance requirements, SecureFiles has introduced scalable and optimized components such as the following:

• Write Gather Cache. To avoid frequent writes during the write operation, the Write Gather Cache component stages the incoming data in memory.

• Background Free Space Monitor. The Background Free Space Monitor manages the use of space for SecureFiles segments by allocating space proactively and reactively.

• In-Memory Space Dispenser. Free space allocations to SecureFiles objects are managed in memory through a shared-memory data structure, part of the system global area known as the In-Memory Space Dispenser.

Overview of SecureFiles Architecture

This section gives a brief overview of the various components of SecureFiles and describes the role of each component during a write operation.

The Delta Update component handles a special case of write, where a small portion of data changes to an existing LOB segment. Located above the Write Gather Cache, the Delta Update component detects whether the incoming data is a list of update deltas (start offset, length of the delta, and the update data). This optimization helps to avoid a rewrite of entire LOB segments.


3

Figure 1. Scalability components in SecureFiles for data manipulation

In most cases, the Write Gather Cache component stages the incoming data in memory. By default, the allocation of this cache is from the SGA and is private to each server process. Data is destaged from this cache on an overflow or at transaction commit. The SecureFiles architecture provides a number of data transformation components under the Write Gather Cache. These components include advanced file system features, such as deduplication, compression, and encryption⎯all of which are optional and governed by the applications at the level of the SecureFiles segment. Oracle 11g documentation includes details of these advanced data transformation components.

After the Write Gather Cache overflows for a transaction or during transaction commits, it requests free space from the space management layer (described below). The space management layer returns an array of chunks (contiguous blocks) that fit the free space request with maximal contiguity.


4

The space management layer consists of an In-Memory Space Dispenser to provide a set of free chunks, and a background free space monitor, to allocate space proactively and reactively.1

Figure 2. Scalability components in SecureFiles for data retrieval operations

After a free space request, the generic database I/O layer is invoked to issue asynchronous I/O to the underlying storage device. Transactions do not commit as long as pending asynchronous I/Os exist. This completes the write operation.

SecureFiles read operations do some prefetching to improve performance. There is noticeable improvement when streaming the entire content of a large SecureFiles segment.

1 Space metadata entries manage space allocation and reuse: committed free space (CFS) and uncommitted free space (UFS). CFS is a set of on-disk space metadata entries for tracking free space; it is usually becomes available from newly allocated extents. Space from CFS transfers to the In-Memory Space Dispenser to service Write Gather Cache demand. UFS is another set of on-drive space metadata entries for tracking free space; it is usually freed by deletes, truncates, and updates. The Background Space Monitor coalesces the free space entries, if possible, and moves them from UFS to CFS blocks if and only if the transactions associated with the generation of these free space entries have committed. The free space entries then transfer to the In-Memory Space Dispensers as and when requested.


5

Also, SecureFiles read operations use a different consistent read (CR) mechanism compared to table CR. SecureFiles does not use undo_tablespace for CR. Instead, updated data remains in the same tablespace, and new allocated chunks contain the changes. Maintained in the metadata are the pointers to updated blocks, and metadata blocks are transactionally managed. As a result, a CR on the metadata blocks provides older versions of SecureFiles segments for CR without blocking write operations.

Hardware Configuration

The hardware used in this performance study was the Sun Storage 6780 array. The hosts consisted of seven four-quad core servers with a minimum of 32 GB of memory, dual-ported 4 Gb Fibre Channel (FC) QLogic Host Bus Adapters (HBAs), with a 1 Gigabit Ethernet interconnect, running Red Hat Enterprise Linux 5.0. Table 1 shows the hardware configuration details.

TABLE 1. HARDWARE CONFIGURATION DETAILS

COMPONENT DESCRIPTION QUANTITY COMMENTS

CPU Quad core Minimum 2 Per server

Memory Minimum 32 GB Per server

HBA QLogic 4 GB Dual port FC, per server

Interconnect Gb 1 Per server

Operating system Linux 5.0 1 Red Hat Enterprise Edition

SAN switch QLogic SAN box 2 Model 5600

Storage Sun Storage 6780 1 With 16 I/O modules, 4 Gb/sec

connectivity and backend connections

Drives 146 GB, FC, 15 K rpm 256 16 drives on each I/O module

There were 15 logical unit numbers (LUNs) created as RAID 0, each using 16 drives across multiple I/O modules. 2 Each LUN was 1 TB. See Appendix A and the “I/O Limits” section for more details.

I/O Limits

The test showed that when using the Oracle ORION utility performing 1 MB write and read, each LUN could drive a maximum throughput of 50 MB/sec for writes and 60 MB/sec for reads.

2 With RAID 10 configuration, seven-node write throughput dropped to 2.5 GB/sec from a RAID 0 level of 4.5 GB/sec. This drop was due to writes to mirror drives consuming almost half of the bandwidth. Read rate stayed the same as the RAID 0 level.


6

Therefore, the maximum write throughput achievable by a single server node was approximately 700 MB/sec (15 LUNs x 50) for random writes of 1 MB or more. Similarly, the maximum read throughput achieved by a server node was approximately 750 MB/sec for random reads of 1 MB or more. The expected result was 2.8 Gb/sec to 3.2 Gb/sec for four-node configurations and 4.9 Gb/sec to 5.2 Gb/sec for seven-node configurations.

Performance Evaluation

Figures 3 and 4 show the benchmark results. The results show a maximum throughput of 700 MB/sec with a single node, 2.8 Gb/sec with four nodes, and 4.5 Gb/sec with seven nodes (90 percent scalability).3

Figure 3 shows the benchmark results, as seen on the host.

Figure 3. SecureFiles figure throughput using one, four, and seven nodes

Figure 4 shows the benchmark results, as seen on the storage array.

Figure 4. Sun Storage 6780 array throughput using one, four, and seven nodes

3 Maximum throughput is achieved with data sizes 1 MB or more.


7

The difference in the performance is due to Oracle database overhead and SecureFiles measurement methodology.

The benchmark tests evaluated SecureFiles performance using the following parameters.

• There was an increase in the number of concurrent users on a single node to evaluate CPU scaling. The test used the following approach:

• For small LOBs, two to five times the number of users for each core

• For medium-to-large LOBs, one to three times the number of users for each core

• There was a variation in SecureFiles data sizes in steps of 10 KB, 100 KB, 1 MB, 10 MB, and 100 MB to evaluate the performance at various data sets. The data set was categorized into four application areas:

• E-mail application data set. Regular e-mail documents ranged from 5 KB to 14 KB in size.

• Document application data set. Microsoft Office and PDF documents ranged from 80 KB to 110 KB.

• Image processing application data set. Low-resolution images ranged from 900 KB to 1.2 MB, and high-resolution bitmap images ranged from 8.5 MB to 11.3 MB.

• Video and movie application data set. The median size of a video was 97 MB, and movies were approximately 4.7 GB.

• There was an increase in the number of server nodes to evaluate scalability, including a scalability test with four nodes and seven nodes.

The Oracle software version used for testing was 11.1.0.7. Each set of tests conducted included 30 seconds of ramp-up, 300 seconds of steady state, and 30 seconds of ramp-down. Measurements were taken during a five-minute steady state. Write and read tests were performed separately. For small write tests, batched commits helped to avoid high log file sync waits. For read tests, each user was provided with a range of primary keys (low value and high value), and selects were randomized within this key range. All the tests used NOCACHE and NOLOGGING storage option settings. The benchmark client driver and server (database) were run on the same machine.

Single-Node Performance

This section describes single-node benchmark numbers at various concurrent user loads along with CPU resource usage. The tests repeated experiments for all four data sets: e-mails, documents, images, and videos.

Table 2 summarizes the total e-mail application data set throughput for 32, 64, and 128 concurrent users (assuming each e-mail message is 10 KB).


8

TABLE 2. E-MAIL DATAS ET - WRITE AND READ RATE

CONCURRENT USERS WRITE THROUGHPUT (MB/SEC) READ THROUGHPUT (MB/SEC)

32 38 65

64 42 90

128 42 90

These numbers translate to a maximum of 15 million e-mail messages inserted and 32 million messages read in an hour, as shown in Figure 5.

Figure 5. E-mail messages processed in millions per hour

As shown in Table 3, CPU resources saturated at 64 concurrent users4⎯and, as shown in Table 2, going to 128 users did not give any additional benefit in throughput.

TABLE 3. E-MAIL DATA SET CPU USAGE − WRITE AND READ

AVERAGE CPU USAGE − WRITE AVERAGE CPU USAGE − READ CONCURRENT

USERS PERCENT USR PERCENT SYS PERCENT USR PERCENT SYS

32 45 25 25 19

64 60 20 60 35

128 65 20 60 35

Document Data Set

Table 4 lists the total document application data set throughput for 32, 64, and 128 users (assuming each document is 100 KB). CPU resource usage is similar to the e-mail data set, so it is not repeated here.

4 CPU usage reported is systemwide, which includes CPU consumption of database server and benchmark driver processes. Analysis assumes that the benchmark driver is lightweighted.


9

TABLE 4. DOCUMENT DATA SET − WRITE AND READ RATE


32 260 260

64 260 320

128 280 320

That is an equivalent of 10 million documents inserted and 16 million documents read per hour, as shown in Figure 6.

Figure 6. Documents processed in millions per hour

Image Data Set⎯Low Resolution

Table 5 shows the total throughput with an image processing application data set, such as low-resolution photos, for 24, 48, and 96 users (assuming each photo is 1 MB).

TABLE 5. IMAGE DATA SET (LOW RESOLUTION) − WRITE AND READ RATE

CONCURRENT USERS WRITE THROUGHPUT

(MB/SEC)

READ THROUGHPUT

(MB/SEC)

24 610 610

48 640 680

96 640 680

With a 1 MB read or write size, throughput was near the maximum bandwidth available in the single-node configuration. This result was not expected from the e-mail and document data sets, because they were of an online transaction processing nature (small random reads/writes), and the system saturation point was determined by the maximum available I/O operations per second (IOPS).

This throughput rate translates to 2.5 million low-resolution images processed in an hour, as shown in Figure 7.


10

Figure 7. Images processed in millions per hour

CPU resources for write and read (as shown in Table 6) saturated with 48 concurrent users. Tests with 96 concurrent users did not achieve any additional throughput, as shown in Table 5.

TABLE 6. IMAGE DATASET CPU USAGE WRITE AND READ

AVERAGE CPU USAGE − WRITE AVERAGE CPU USAGE − READ CONCURRENT

USERS PERCENT USR PERCENT SYS PERCENT USR PERCENT SYS

24 66 23 50 27

48 75 25 65 30

96 76 23 69 28

Image Data Set⎯High Resolution

Table 7 shows the total throughput for an image processing application data set involving high-resolution images, such as RAW photos and X-rays, for 12, 24, and 48 users (assuming each image is 10 MB). This result translates to 250,000 images processed in an hour.

CPU usage is not repeated because it is similar to the low-resolution image data set (as shown in Table 6).

TABLE 7. IMAGE DATA SET (HIGH RESOLUTION) − WRITE AND READ RATE


12 640 620

24 690 700

48 690 700

Video Data Set

Table 8 summarizes the total video application dataset throughput for 8, 16, and 32 concurrent users (assuming each video file is 100 MB).


11

TABLE 8. VIDEO DATA SET − WRITE AND READ RATE


8 620 650

16 690 700

32 690 700

This translates to a maximum of 25,000 video files processed in an hour, as shown in Figure 8.

Figure 8. Videos processed in thousands per hour

It is also clear from throughput results (Tables 2, 4, 5, 7, and 8) that the system is saturated with fewer users as the data size increases.

CPU usage is similar to Table 6, so it is omitted.

Multiple-Node Performance

This section describes the performance and scalability when running SecureFiles on a four- to seven-node configuration. Based on a single-node throughput saturation point with respect to concurrent users, multiple-node benchmarks were limited to one set of concurrent user tests for each data set. For example, in a single-node e-mail application data set benchmark, because the system was saturated at 64 concurrent users, benchmarks were conducted only with 64 users for the e-mail data set in a four- to seven-node configuration, similarly for other data sets.

Figure 9 shows the total write throughput for various data sets using one node, four nodes, and seven nodes. The maximum throughput for a four-node configuration is 2.8 Gb/sec and 4.5 Gb/sec for a seven-node configuration, with a scalability factor of 90 percent on both configurations.5

5 To achieve this scalability and avoid space metadata contention, range-hash partitioning based on virtual columns is used.


12

Figure 9. Write scalability across one node, four nodes, and seven nodes

Figure 10 shows the total read throughput for various data sets using one node, four nodes, and seven nodes.

Figure 10. Read scalability across one node, four nodes, and seven nodes

TABLE 9. SEVEN NODE RESULTS − OPERATIONS PER HOUR

DATASHEET OPERATIONS PER HOUR

E-mail 171 million messages/hr.

Documents 92 million documents/hr.

Low-resolution images 16 million images/hr.

High-resolution images 1.6 million images/hr.

Videos 160 KB video files/hr.


13

Performance Guidelines

There are two categories of performance guidelines:

• A list of general recommendations made without regard to SecureFiles size

• A list of recommendations based on SecureFiles size that takes into account the nature of the application (write intensive or read intensive)

General Recommendations

For best performance with less effort, Oracle advises LOB storage settings of NOCACHE and NOLOGGING.

• The NOCACHE setting bypasses the buffer cache, and the server process does direct path read/write (using ASYNC I/O).

• The NOLOGGING setting avoids redo generation for LOB segments (redo for metadata and table data is still generated). This setting has database recoverability implications. Refer to Oracle documentation for details.

• If the LOGGING setting is enabled, place redo log files in the highest-performing set of drives. Also, increase the log_buffer setting if the “log buffer space” wait is seen.

• The CACHE READS option can be used if the same LOB segment is read frequently. Size db_buffer_cache accordingly so that LOB data does not flush other cached objects.

• Heavy data manipulation language activity can cause space metadata contention. Using a large number of hash partitions can spread this contention.

• For other performance considerations, refer to the white papers listed in Appendix C.

• Intensive application performance tuning best practice is to keep the average response time of drive read or writes under 10 ms. An Oracle Automatic Workload Repository report can provide details.

As shown in Table 10, an average wait is approximately 8 ms, which is good.

TABLE 10 TYPICAL ORACLE AUTOMATIC WORKLOAD REPOSITORY TOP WAITS FOR WRITE-INTENSIVE APPLICATIONS

EVENT WAITS TIMES(S) AVG WAIT (MS) PERCENT DB TIME

direct path write 460,558 3,655 8 36.0

DB CPU 3,168 31.2

SQL*Net more data from client 6,660,164 2,775 0 27.4

db file sequential read 15,480 131 8 1.3

gc buffer busy acquire 9,442 124 13 1.2


14

As shown in Table 11, an average wait is approximately 15 ms, slightly above the performance cutoff of 10 ms.

TABLE 11. TYPICAL ORACLE AUTOMATIC WORKLOAD REPOSITORY TOP WAITS FOR READ-INTENSIVE APPLICATIONS

WAITS TIME(S) AVG WAIT (MS) PERCENT DB TIME

direct path write 241,071 3,647 15 45.0

DB CPU 2,597 32.0

SQL*Net more data from client 6,915,542 1,760 0 21.7

db file sequential read 779 36 47 .4

gc buffer busy acquire 195 29 148 .4

Recommendations Based on SecureFiles Size

Depending on the size of the SecureFiles being stored, follow these recommendations in setting up your system configuration.

Very Small SecureFiles (4 KB−100 KB)

For very small SecureFiles, use the following recommended setup:6

• Application areas. E-mail messages, HTML pages.

• Write-intensive applications. Write-intensive applications in this size range demand high IOPS, so you can configure the system for maximum IOPS. At a high level of concurrency, users might notice log file sync waits. To reduce this wait, follow general tuning practice, such as placing redo log files on the highest-performing set of drives, or use batch commits.

• Read-intensive applications. Configure the system for maximum IOPS. Storage option CACHE READS can be tried if the same data is frequently accessed.

Small SecureFiles (100 KB−1 MB)

For small SecureFiles, use the following recommended setup:

• Application areas. Office and PDF documents, audio files, low-quality images, and photos.

• Write-intensive applications. When reaching approximately 1 M file sizes, I/O becomes more efficient and can saturate available I/O bandwidth.

6 Usually, performance of SecureFiles in this size range is compared to the LONG or VARCHAR2 datatypes. Note that these datatypes are stored in regular table segment. In addition, read and write operations use db_buffer_cache; so for fair comparison, use SecureFiles with the CACHE option.


15

• Read-intensive applications. Less or no contention. Pay attention to SQL*Net traffic. Usually the network interface between the client and the server becomes the bottleneck.7

Medium and Large SecureFiles (More than 1 MB)

For medium and large SecureFiles, use the following recommended setup:

• Application areas. High-resolution images, such as X-rays, videos, and movies.

• Write-intensive applications. Less or no contention, usually CPU, I/O, and network bandwidth is the limiting factor.

• Read-intensive applications. No contention, usually CPU, I/O, and network bandwidth is the limiting factor.

Conclusion

These breakthrough results using Oracle Database 11g for file system storage and the Sun Storage 6780 array prove that SecureFiles provides performance that is far greater than traditional file systems. This approach delivers superior scalability, optimizes the use of the database for storage of all enterprise data (structured and unstructured), and offers other advanced database capabilities that are not possible with conventional file systems.

The SecureFiles feature in Oracle Database 11g—combined with Oracle Real Application Clusters, Oracle Automatic Storage Management, partitioning, security, and information lifecycle management—provides customers with superior performance, scalability, high availability, and control over structured and unstructured content.

7 A standard 10/100 network interface card (NIC) provides 20 MB/sec, whereas a gigabit NIC provides about 80 MB/sec to120 MB/sec of transfer rate. Often these specifications are misinterpreted as gigabytes⎯off by a factor of eight.


16

Appendix A: Hardware Setup

As a reference architecture for SecureFiles, Oracle’s Sun Storage 6780 array is capable of supporting both high-throughput as well as high-transaction rates. The I/O architecture of the controller provides the performance necessary for high-sustained throughput as data is pulled from the drives. This particular configuration was done with a 4 Gb/sec Fibre Channel (FC) interconnect infrastructure and 256 drives. The FC infrastructure now supports 8 Gb/sec interfaces, and the drive count nearly doubles to 448 drives. With upgrades to the rest of the infrastructure, an increase in performance is a reasonable expectation.

Figure A1. Sample architectural layout depicting the Sun Storage 6780 array


17

Figure A2. Array LUN configuration for SecureFiles


18

Appendix B: Oracle Setup and Test Description

Oracle version 11.1.0.7 is used in this benchmark, with Oracle Automatic Storage Management, partitioning, and Oracle Real Application Clusters options.

Init.ora

*.db_files=1024

*.db_name=’secfiles’

*.db_block_size=8192

*.compatible=’11.1.0.0.0’

*.control_files=’+DATA/secfiles/controlfile/ Current.271.682862241’

*.db_cache_size=4G

*.shared_pool_size=2048M

*.java_pool_size=128M

*.large_pool_size=128M

*.log_buffer=67108864

*.pga_aggregate_target=4G

*.processes=500

*.sessions=500

*.open_cursors=500

*.audit_trail=’none’

*.cluster_database=true

*.cluster_database_instances=7

*.log_checkpoint_interval=100000000

*.log_checkpoint_timeout=100000000

*.log_checkpoints_to_alert=TRUE


19

*.db_block_checking=FALSE

*.db_block_checksum=FALSE

*._transaction_auditing=FALSE

*._library_cache_advice=FALSE

*._resource_manager_always_off=TRUE

*._fast_cursor_reexecute=true

DDL

CREATE TABLE doc_tab

(pkey NUMBER(10) not null, document BLOB,

pkey_virtual number(1) generated always as

(to_number(substr(to_char(pkey), 1,1))))

LOB (document) store as securefile doc_lob_seg (NOCACHE

NOLOGGING enable storage in row)

tablespace demo_tst

partition by range (pkey_virtual)

subpartition by hash (pkey) subpartitions 31

(partition pkey_v1 values less than (2) lob(document)

(tablespace demo_tsl_p1) ,

partition pkey_v2 values less than (3) lob(document)

(tablespace demo_tsl_p2),











20


partition pkey_v_gt_7 values less than (MAXVALUE)

lob(document) (tablespace demo_tsl_max) );

CREATE UNIQUE INDEX pkey_idx ON doc_tab (pkey, pkey_virtual)

local tablespace idx_ts compute statistics;

Appendix C: White Papers

• oracle.com/technology/products/database/securefiles/pdf/securefilesperformancepaper.pdf

• oracle.com/technology/products/database/securefiles/pdf/securefiles.pdf

Using SecureFiles on a Sun Storage 6780 Array March 2010 Oracle Corporation World Headquarters 500 Oracle Parkway Redwood Shores, CA 94065 U.S.A. Worldwide Inquiries: Phone: +1.650.506.7000 Fax: +1.650.506.7200 oracle.com

Copyright 2009, 2010, Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only and the contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or fitness for a particular purpose. We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose, without our prior written permission. Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners. AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. UNIX is a registered trademark licensed through X/Open Company, Ltd. 0110

Using SecureFiles on a Sun Storage 6780 Array...Oracle White Paper—Using SecureFiles on a Sun...

Documents

Transcript of Using SecureFiles on a Sun Storage 6780 Array...Oracle White Paper—Using SecureFiles on a Sun...