Deploy Large Unstructured Data Storage in a Small … · Red Hat Gluster Storage is open,...
-
Upload
truongkien -
Category
Documents
-
view
216 -
download
0
Transcript of Deploy Large Unstructured Data Storage in a Small … · Red Hat Gluster Storage is open,...
© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 1 of 25
White Paper
Deploy Large Unstructured Data Storage in a Small Data Center Space: Cisco UCS C240 M4 Rack Server and Red Hat Gluster Storage 100-TB Solution
Executive Summary
Today, companies face scenarios that require an IT architecture that can hold tens to hundreds of terabytes of
unstructured data. Storage-intensive enterprise workloads can encompass the following:
● Archive and backup content, including backup images and near-online (nearline) archives
● Rich media content storage and delivery, such as videos, images, and audio files
● Enterprise drop-box content
● Cloud and business applications, including log files, and RFID and other machine-generated data
● Virtual and cloud infrastructure, such as virtual machine images
● Emerging workloads, such as co-resident applications
This document is intended to help organizations seeking an ultra-dense, high-throughput solution that can store up
to 100-TB of unstructured data in a small amount of rack space. The document provides test results that
demonstrate how Cisco UCS® C240 M4 Rack Servers and Red Hat Gluster Storage, along with Cisco Unified
Computing System™
(Cisco UCS) fabric interconnects, can be optimized to serve in these scenarios. It includes
system setup information, test methodology, and results for a 100-terabyte (TB) solution. Initial tests indicate that
the Cisco® and Red Hat technologies together provide an ultra-dense, scalable, high-throughput solution that can
store unstructured data that is easy to manage.
Solution Overview
The Cisco UCS C240 M4 Rack Server and Red Hat Gluster Storage provide a complete 100-TB solution for high-
volume, unstructured data storage.
Cisco UCS C240 M4 Rack Server
The Cisco UCS C240 M4 large form-factor (LFF) server is the newest 2-socket, 2-rack-unit (2RU) rack server from
Cisco. It is designed for both performance and expandability over a wide range of storage-intensive infrastructure
workloads, from big data to collaboration.
The enterprise-class Cisco UCS C240 M4 small form-factor (SFF) server extends the capabilities of the Cisco UCS
portfolio in a 2RU form factor with the addition of the Intel® Xeon
® processor E5-2600 v3 series, which delivers the
best combination of performance, flexibility, and efficiency gains.
In addition, the Cisco UCS C240 M4 LFF server provides 24 DIMM slots, up to 6 PCI Express (PCIe) 3.0 slots, and
up to 12 front-loading LFF drives plus 2 (optional) internal SFF SATA boot drives for a total of 14 internal drives.
© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 2 of 25
The Cisco UCS C240 M4 server includes a modular LAN-on-motherboard (mLOM) slot for installation of a Cisco
virtual interface card (VIC) or third-party network interface card (NIC) without consuming a PCI slot; it also includes
two 1 Gigabit Ethernet embedded (on the motherboard) LOM ports. These features together provide an
outstanding amount of internal memory and storage expandability plus exceptional performance.
The Cisco UCS C240 M4 can be used as a standalone server or as part of the Cisco Unified Computing System,
which unifies computing, networking, management, virtualization, and storage access into a single integrated
architecture enabling end-to-end server visibility, management, and control in both bare-metal and virtualized
environments.
Cisco UCS 6200 Series Fabric Interconnects
The Cisco UCS 6200 Series Fabric Interconnects are a core part of Cisco UCS, providing both network
connectivity and management capabilities for the system. The Cisco UCS 6200 Series offers line-rate, low-latency,
lossless 10 Gigabit Ethernet, Fibre Channel over Ethernet (FCoE), and Fibre Channel functions.
The Cisco UCS 6200 Series provides the management and communications backbone for the Cisco UCS Servers
and 5100 Series Blade Server Chassis. All chassis, and therefore all blades and rack servers, attached to the
Cisco UCS 6200 Series Fabric Interconnects become part of a single, highly available management domain. In
addition, by supporting unified fabric, the fabric interconnect provides both the LAN and SAN connectivity for all
blades within its domain.
The Cisco UCS 6200 Series uses a cut-through networking architecture, supporting deterministic, low-latency, line-
rate 10 Gigabit Ethernet on all ports, switching capacity of 2 terabits (Tb), and 320 Gbps of bandwidth per chassis,
independent of packet size and enabled services. The product family supports Cisco low-latency, lossless 10
Gigabit Ethernet unified network fabric capabilities, which increase the reliability, efficiency, and scalability of
Ethernet networks. The fabric interconnect supports multiple traffic classes over a lossless Ethernet fabric from the
blade through the interconnect. Significant savings in total cost of ownership (TCO) come from an FCoE-optimized
server design in which NICs, host bus adapters (HBAs), cables, and switches can be consolidated.
Red Hat Gluster Storage
Red Hat Gluster Storage is open, software-defined scale-out storage that easily manages unstructured data for
physical, virtual, and cloud environments. Combining both file and object storage with a scale-out architecture
(Figure 1), it is designed to cost-effectively store and manage petabyte-scale data even as the amount of data
continues to grow. It also delivers a continuous storage fabric across physical, virtual, and cloud resources so that
organizations can transform their big, semistructured, and unstructured data from a burden into an asset.
© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 3 of 25
Figure 1. Red Hat Gluster Storage Scales Up and Out
Built on the industry-leading Red Hat Enterprise Linux (RHEL) operating system, Gluster Storage offers cost-
effective and highly available storage without scale or performance compromises (Table 1). Organizations can use
it to avoid storage silos by enabling global access to data through multiple file and object protocols. And it works
transparently with Cisco UCS C240 M4 servers.
Table 1. Red Hat Gluster Storage Features
Single global namespace Aggregates disk and memory resources into a single trusted storage pool
Replication Supports synchronous replication within a data center and asynchronous replication for disaster recovery
Snapshots Helps ensure data protection through clusterwide filesystem snapshots that are user accessible for easy recovery of files
Elastic hashing algorithm Eliminates performance bottlenecks and single points of failure because it does not have any metadata server layer
Easy online management ● Web-based management console
● Powerful and intuitive command-line interface (CLI) for Linux administrators
● Monitoring (Nagios based)
● Capability to expand and shrink storage capacity without downtime
Industry-standard client support ● Network File System (NFS), Server Message Block (SMB) for file-based access
● OpenStack Swift for object access
● GlusterFS native client for highly parallelized access
System Specifications
This 100-TB solution includes Cisco UCS C240 M4 Rack Servers, Red Hat Gluster Storage, and Cisco UCS
6248UP 48-Port Fabric Interconnects. For the purpose of running the performance tests, the components of the
solution were configured to the specifications described in this section. Figure 2 provides an overview of the
system.
© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 4 of 25
Figure 2. Solution Diagram
Component Configuration
Table 2 describes how the components of the solution were configured for the tests.
Table 2. Component Configuration
Component Configuration
Cisco UCS C240 M4 Rack Server 4 Cisco UCS C240 M4 servers, each configured with
● 12 x 6-TB 7200-rpm disks
● 2 Intel Xeon processor E5-2660 v3 CPUs
● 256 GB of RAM (sixteen 16-GB DDR4 at 2133 MHz)
● 1 Cisco UCS VIC 1227 dual-port 10-Gbps Enhanced Small Form-Factor Pluggable (SFP+)
● 1 Cisco 12-Gbps SAS RAID controller with a 4-GB flash-backed write cache (FBWC)
● 2 internal 2.5-inch SATA solid-state disk (SSD) drives
Operating system Red Hat Enterprise Linux 6.6
Red Hat Gluster Storage software Red Hat Gluster Storage 3.0.4
Connectivity 2 Cisco UCS 6248UP 48-Port Fabric Interconnects supporting 1- and 10-Gbps SFP+
RAID Configuration
The RAID configuration includes the following:
● Each Cisco UCS C240 M4 server includes 12-disk RAID 6 arrays for the Gluster Storage bricks, created
using the LSI RAID controller graphical configuration utility.
● The RAID controller cache is set to write back with battery-backed write cache.
● Each RAID 6 array includes one virtual disk (60 TB), and each virtual disk has a capacity of 55 TB after
formatting with XFS.
Appendix A shows the StorCLI output results.
Network Configuration
The network is configured as follows:
● Each Cisco UCS C240 M4 server includes a Cisco UCS VIC 1227 adapter.
© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 5 of 25
● Each VIC 1227 adapter has two 10-Gbps ports, with one port connected to fabric interconnect A and the
other port connected to fabric interconnect B. With this configuration, each Cisco UCS C240 M4 has an
active-passive 2 x 10-Gbps connection to each fabric interconnect.
Red Hat Gluster Storage Installation and Configuration
Installation and configuration of Red Hat Gluster Storage consists of several steps:
1. Install the Red Hat Gluster Storage 3.0.4 ISO image on one of the two internal SSDs in the Cisco UCS C240
M4 servers. This release is based on RHEL 6.6 and GlusterFS 3.6. For step-by-step installation and
configuration guidance for Gluster Storage, visit https://access.redhat.com/site/documentation/en-
US/Red_Hat_Storage/.
2. Install RHEL 7.1 on the client servers, following instructions in the administration guide for installing the native
client in Red Hat Gluster Storage.
3. Create storage bricks, using the rhs-server-init.sh script on all nodes.
4. Make sure that the glusterd daemon is running on each of the Red Hat Gluster Storage servers (gluster5,
gluster6, gluster7, and gluster8), and then use the gluster peer probe command to create the trusted storage
cluster from the gluster1 server:
# gluster peer probe gluster6
# gluster peer probe gluster7
# gluster peer probe gluster8
5. Confirm that all the storage servers are in a connected state using the gluster peer status command:
# gluster peer status
6. Create the distributed replicated (two-way) volume from the gluster1 server. (See Appendix B for the full
volume-creation script.)
7. Start the volume, and look at the status:
The rhs-server-init.sh script does the following:
● Creates a physical volume.
● Creates a logical volume.
● Places the XFS file system on the logical volume.
● Runs a tuned performance profile for Red Hat Gluster Storage virtualization.
© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 6 of 25
# gluster volume start gv0
# gluster volume info
# gluster volume status
8. Confirm that all the storage servers are in a connected state.
9. Mount the GlusterFS volume gvol0 on client 1:
# mkdir /mnt/gv0
# mount -t glusterfs gluster1:/gv0 /mnt/gv0
10. Repeat these commands on the other 15 clients.
Note: Even though the same Red Hat Gluster Storage server is being used for mounting the volumes, the native
client protocol has built-in load balancing. The clients use the mount server initially to get the volume information,
and after that they contact the individual storage servers directly to access the data. The data request does not
have to go through the mount server.
© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 7 of 25
Test Methodology and Results
The test environment was set up to include industry-standard tools to measure read-write performance and to
provide benchmarking for performance across a specific cluster and for performance of distributed storage.
Driver Setup
Eight Cisco UCS B200 M4 servers were used to transfer I/O to the 100-TB Gluster Storage cluster. Each server
was equipped as follows:
● Two Intel Xeon processor E5-2600 series CPUs
● 256 GB of RAM
● One Cisco VIC 1240 capable of 40 Gbps of throughput
The Cisco UCS 5100 Series chassis housing the Cisco UCS B200 M4 servers includes two 8-port Cisco UCS
2208XP Fabric Extender I/O modules connected to the two Cisco UCS 6248UP fabric interconnects.
To transfer I/O, each Cisco UCS B200 M4 server also ran a hypervisor and two virtual switches, with a dedicated
10 Gigabit Ethernet network interface. Each virtual machine was running RHEL 7.1 and the I/O benchmark tools
discussed in the following sections.
Sequential I/O Benchmarking: IOzone Tool
IOzone is a file-system benchmark tool that is useful for performing a broad file-system analysis of a computer
platform. In this case, IOzone was used to test the sequential read-write performance of the GlusterFS replicated
volume. IOzone’s cluster mode option -+m is particularly well-suited for distributed storage testing because testers
can start many worker threads from a variety of client systems in parallel, targeting the GlusterFS volume. The 2x2
distributed replicated volume was tested using the following command with a total of 128 threads running across 16
client systems (8 threads per client):
# iozone -+m ${IOZONE_CONFIG_FILENAME} -i ${IOZONE_TEST} -C -w -+n -s ${IOZONE_FILESZ} -r
${IOZONE_RECORDSZ} -+z -c -e -t ${TEST_THREADS}
The following parameters were used in the IOzone command line:
● -+m specifies the cluster testing mode.
● IOZONE_CONFIG_ FILENAME is the IOzone configuration file for the cluster mode. The file lists the client
host names, the associated GlusterFS mount point, and the location of the iozone utility.
● The IOZONE_TEST parameter varied to cover the sequential-read and sequential-write test cases.
● IOZONE_FILESZ is set to 8 GB. 8-GB file transfers are used as representative of the workload for the
content cloud reference architecture testing.
● IOZONE_RECORDSZ varied between 64 KB and 16 MB, using record sizes that were powers of two. This
range of record sizes is intended to characterize the effect of the record or block size (in client file requests)
on I/O performance.
● TEST_THREADS specifies the number of threads or processes that are active during the measurement.
128 threads are used across 16 client systems (8 threads per client).
Figure 3 shows sequential read throughput of up to 3.8 GBps and up to 2.3-GBps replicated write results for typical
sequential I/O sizes.
© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 8 of 25
Figure 3. Sequential Throughput: IOzone Tool
Performance Benchmarking Across a Cluster: SmallFile Tool
The SmallFile benchmark is a Python-based small-file distributed POSIX workload generator that can be used to
measure performance for a variety of small-file and file metadata-intensive workloads across an entire cluster. The
SmallFile benchmark was used to complement the IOzone benchmark, which is primarily used for measuring the
performance of large-file workloads. Although the SmallFile benchmark kit supports multiple operations, only the
create operation was used to create files and write data to them using varying file sizes ranging from 10 KB to
2 GB.
SmallFile benchmark parameters included the following:
● The create operation is used to create a file and write data to it.
● Eight threads are running on each of the 16 clients (client 1 through client 16).
● The file size is 10 KB, and each thread processes 100,000 files.
● Each thread pauses for 10 microseconds before starting the next file operation, and the response time is
the file operation duration, measured to microsecond resolution.
● For files larger than 1 MB, a record size of 1024 KB is used to determine how much data is transferred in a
single write system call.
The SmallFile command line is as follows:
# python /root/smallfile_cli.py --operation create --threads 8 --file-size
10 --files 100000 --top $path --network-sync-dir /mnt/lockvol/smf-shared
--pause 10 --host-set client1,client2 … client16 --response-times Y
© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 9 of 25
The benchmark returns the number of files processed per second, and it returns the rate that the application
transferred data in MBps.
SmallFile test results show that up to 1530 tiny files per second were created simultaneously, and that write
throughput for the larger files was up to 2.1 GBps (Figure 4). The graph in Figure 4 reflects the trade-off in
throughput (files per second) and transfer rate (MBps) when transferring small files compared large files.
Figure 4. File Create Operations: SmallFile Tool
Random I/O Performance: Flexible I/O Tool
Flexible I/O (FIO) is an industry-standard storage performance benchmark that has been used primarily for single-
server testing. However, with the recent addition of client and server options, it can also be used in a distributed
storage environment. The front end and back end of FIO can be run separately, with the FIO server generating an
I/O workload on the system under test while being controlled from another system. The --server option is used to
launch FIO in a request listening mode. The --client option is used to pass a list of test servers along with the FIO
profile to define the workload to be generated.
FIO was used to determine the random I/O performance using smaller block sizes (ranging from 4 to 32 KB).
Appendix C lists the FIO profile used on each client driver for the random test.
The cluster accommodated about 3000 random read I/O operations per second (IOPS) at typical sizes with a
latency of about 11 milliseconds (ms), as shown in Figure 5.
© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 10 of 25
Figure 5. Random Read Tests: FIO Tool
The solution achieves about 2000 random write IOPS at typical sizes, as shown in Figure 6. Latency is less than
18 ms using the write cache on the array controllers on the Cisco UCS C3160 Rack Server.
Figure 6. Random Write Tests: FIO Tool
© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 11 of 25
Conclusion
Companies facing scenarios that require support for hundreds of terabytes of unstructured data can use a
combination of Cisco UCS C-Series and Red Hat Gluster Storage 100-TB technologies to create a scalable, high-
throughput solution that can meet these demands while remaining easy to manage. Test results using FIO,
SmallFile, and IOzone show remarkable performance and indicate that this solution can handle storage-intensive
enterprise workloads in a dense data center footprint.
For More Information
● Cisco UCS C240 M4 LFF Rack Server
● Cisco UCS 6200 Series Fabric Interconnects
● Red Hat Gluster Storage
● IOzone test
● Large Unstructured Data Storage in a Small Datacenter Footprint: Cisco UCS C3160 and Red Hat Gluster
500-TB Solution
Appendix A: StorCLI Output
[root@gluster5 ~]# /opt/MegaRAID/storcli/storcli64 /c0 show all
Generating detailed summary of the adapter, it may take a while to complete.
Controller = 0
Status = Success
Description = None
Basics :
======
Controller = 0
Model = Cisco 12G SAS Modular Raid Controller
Serial Number = SV50239847
Current Controller Date/Time = 10/07/2015, 15:38:41
Current System Date/time = 10/07/2015, 08:38:43
SAS Address = 578da6e715b66710
PCI Address = 00:0a:00:00
Mfg Date = 01/09/15
Rework Date = 00/00/00
Revision No = 05C
Version :
=======
Firmware Package Build = 24.5.0-0020
Firmware Version = 4.250.00-3632
© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 12 of 25
Bios Version = 6.19.05.0_4.16.08.00_0x06080500
NVDATA Version = 3.1406.00-0079
Ctrl-R Version = 5.06-0004
Boot Block Version = 3.02.00.00-0001
Driver Name = megaraid_sas
Driver Version = 06.803.01.00-rh1
Bus :
===
Vendor Id = 0x1000
Device Id = 0x5D
SubVendor Id = 0x1137
SubDevice Id = 0xDB
Host Interface = PCIE
Device Interface = SAS-12G
Bus Number = 10
Device Number = 0
Function Number = 0
Pending Images in Flash :
=======================
Image name = No pending images
Status :
======
Controller Status = Optimal
Memory Correctable Errors = 0
Memory Uncorrectable Errors = 0
ECC Bucket Count = 0
Any Offline VD Cache Preserved = No
BBU Status = 0
Support PD Firmware Download = No
Lock Key Assigned = No
Failed to get lock key on bootup = No
Lock key has not been backed up = No
Bios was not detected during boot = No
Controller must be rebooted to complete security operation = No
A rollback operation is in progress = No
At least one PFK exists in NVRAM = No
SSC Policy is WB = No
Controller has booted into safe mode = No
© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 13 of 25
Supported Adapter Operations :
============================
Rebuild Rate = Yes
CC Rate = Yes
BGI Rate = Yes
Reconstruct Rate = Yes
Patrol Read Rate = Yes
Alarm Control = Yes
Cluster Support = No
BBU = Yes
Spanning = Yes
Dedicated Hot Spare = Yes
Revertible Hot Spares = Yes
Foreign Config Import = Yes
Self Diagnostic = Yes
Allow Mixed Redundancy on Array = No
Global Hot Spares = Yes
Deny SCSI Passthrough = No
Deny SMP Passthrough = No
Deny STP Passthrough = No
Support more than 8 Phys = Yes
FW and Event Time in GMT = No
Support Enhanced Foreign Import = Yes
Support Enclosure Enumeration = Yes
Support Allowed Operations = Yes
Abort CC on Error = Yes
Support Multipath = Yes
Support Odd & Even Drive count in RAID1E = No
Support Security = Yes
Support Config Page Model = Yes
Support the OCE without adding drives = Yes
Support EKM = Yes
Snapshot Enabled = No
Support PFK = Yes
Support PI = Yes
Support LDPI Type1 = No
Support LDPI Type2 = No
Support LDPI Type3 = No
Support Ld BBM Info = No
Support Shield State = Yes
Block SSD Write Disk Cache Change = Yes
Support Suspend Resume BG ops = Yes
Support Emergency Spares = No
Support Set Link Speed = Yes
© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 14 of 25
Support Boot Time PFK Change = No
Support JBOD = Yes
Disable Online PFK Change = No
Support Perf Tuning = Yes
Support SSD PatrolRead = Yes
Real Time Scheduler = Yes
Support Reset Now = Yes
Support Emulated Drives = Yes
Headless Mode = Yes
Dedicated HotSpares Limited = No
Point In Time Progress = Yes
Extended LD = No
Boot Volume Supported = No
Support Uneven span = No
Support Config Auto Balance = No
Support Maintenance Mode = No
Support Diagnostic results = Yes
Support Ext Enclosure = No
Support Sesmonitoring = No
Support SecurityonJBOD = No
Support ForceFlash = No
Support DisableImmediateIO = No
Support DrvActivityLEDSetting = No
Support FlushWriteVerify = No
Support CPLDUpdate = No
Support ForceTo512e = Yes
Support discardCacheDuringLDDelete = No
Support JBOD Write cache = No
Supported PD Operations :
=======================
Force Online = Yes
Force Offline = Yes
Force Rebuild = Yes
Deny Force Failed = No
Deny Force Good/Bad = No
Deny Missing Replace = No
Deny Clear = No
Deny Locate = No
Support Power State = Yes
Set Power State For Cfg = No
Support T10 Power State = No
Support Temperature = Yes
NCQ = Yes
© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 15 of 25
Support Max Rate SATA = No
Supported VD Operations :
=======================
Read Policy = Yes
Write Policy = Yes
IO Policy = Yes
Access Policy = Yes
Disk Cache Policy = Yes
Reconstruction = Yes
Deny Locate = No
Deny CC = No
Allow Ctrl Encryption = No
Enable LDBBM = No
Support FastPath = Yes
Performance Metrics = Yes
Power Savings = No
Support Powersave Max With Cache = No
Support Breakmirror = Yes
Support SSC WriteBack = No
Support SSC Association = Yes
Support VD Hide = No
Support VD Cachebypass = No
Support VD discardCacheDuringLDDelete = No
Advanced Software Option :
========================
-----------------------------------------------------
Adv S/W Opt Time Remaining Mode
-----------------------------------------------------
MegaRAID FastPath Unlimited Factory Installed
MegaRAID SafeStore Unlimited Secured
MegaRAID RAID6 Unlimited Secured
MegaRAID RAID5 Unlimited Secured
-----------------------------------------------------
Safe ID = XS66VHB2XEGSM7VFM4D5IUCJ473SFKCDW4GWIW2Z
HwCfg :
=====
ChipRevision = C0
BatteryFRU = N/A
© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 16 of 25
Front End Port Count = 0
Backend Port Count = 8
BBU = Present
Alarm = On
Serial Debugger = Present
NVRAM Size = 32KB
Flash Size = 32MB
On Board Memory Size = 4095MB
CacheVault Flash Size = 16.0 GB
TPM = Absent
Upgrade Key = Present
On Board Expander = Absent
Temperature Sensor for ROC = Present
Temperature Sensor for Controller = Absent
Upgradable CPLD = Absent
Current Size of CacheCade (GB) = 0
Current Size of FW Cache (MB) = 3534
ROC temperature(Degree Celsius) = 58
Policies :
========
Policies Table :
==============
------------------------------------------------
Policy Current Default
------------------------------------------------
Predictive Fail Poll Interval 300 sec
Interrupt Throttle Active Count 16
Interrupt Throttle Completion 50 us
Rebuild Rate 30 % 30%
PR Rate 30 % 30%
BGI Rate 30 % 30%
Check Consistency Rate 30 % 30%
Reconstruction Rate 30 % 30%
Cache Flush Interval 4s
------------------------------------------------
Flush Time(Default) = 4s
Drive Coercion Mode = 1GB
Auto Rebuild = On
Battery Warning = On
ECC Bucket Size = 15
© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 17 of 25
ECC Bucket Leak Rate (hrs) = 24
Restore HotSpare on Insertion = Off
Expose Enclosure Devices = On
Maintain PD Fail History = Off
Reorder Host Requests = On
Auto detect BackPlane = SGPIO/i2c SEP
Load Balance Mode = Auto
Security Key Assigned = Off
Disable Online Controller Reset = Off
Use drive activity for locate = Off
Boot :
====
BIOS Enumerate VDs = 1
Stop BIOS on Error = Off
Delay during POST = 0
Spin Down Mode = None
Enable Ctrl-R = Yes
Enable Web BIOS = No
Enable PreBoot CLI = No
Enable BIOS = Yes
Max Drives to Spinup at One Time = 2
Maximum number of direct attached drives to spin up in 1 min = 10
Delay Among Spinup Groups (sec) = 12
Allow Boot with Preserved Cache = Off
High Availability :
=================
Topology Type = None
Cluster Permitted = No
Cluster Active = No
Defaults :
========
Phy Polarity = 0
Phy PolaritySplit = 0
Strip Size = 64kB
Write Policy = WT
Read Policy = None
Cache When BBU Bad = Off
Cached IO = Off
VD PowerSave Policy = Controller Defined
© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 18 of 25
Default spin down time (mins) = 30
Coercion Mode = 1 GB
ZCR Config = Unknown
Max Chained Enclosures = 2
Direct PD Mapping = No
Restore Hot Spare on Insertion = No
Expose Enclosure Devices = Yes
Maintain PD Fail History = No
Zero Based Enclosure Enumeration = No
Disable Puncturing = No
EnableLDBBM = No
DisableHII = No
Un-Certified Hard Disk Drives = Allow
SMART Mode = Mode 6
Enable LED Header = No
LED Show Drive Activity = No
Dirty LED Shows Drive Activity = No
EnableCrashDump = No
Disable Online Controller Reset = No
Treat Single span R1E as R10 = No
Power Saving option = Enabled
TTY Log In Flash = Yes
Auto Enhanced Import = Yes
BreakMirror RAID Support = Yes
Disable Join Mirror = No
Enable Shield State = Yes
Time taken to detect CME = 60 sec
Capabilities :
============
Supported Drives = SAS, SATA
Boot Volume Supported = NO
RAID Level Supported = RAID0, RAID1, RAID5, RAID6, RAID00, RAID10, RAID50,
RAID60, PRL 11, PRL 11 with spanning, SRL 3 supported,
PRL11-RLQ0 DDF layout with no span, PRL11-RLQ0 DDF layout with span
Enable JBOD = No
Mix in Enclosure = Allowed
Mix of SAS/SATA of HDD type in VD = Allowed
Mix of SAS/SATA of SSD type in VD = Allowed
Mix of SSD/HDD in VD = Not Allowed
SAS Disable = No
Max Arms Per VD = 32
Max Spans Per VD = 8
Max Arrays = 128
© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 19 of 25
Max VD per array = 64
Max Number of VDs = 64
Max Parallel Commands = 928
Max SGE Count = 60
Max Data Transfer Size = 8192 sectors
Max Strips PerIO = 42
Max Configurable CacheCade Size(GB) = 0
Min Strip Size = 64 KB
Max Strip Size = 1.0 MB
Scheduled Tasks :
===============
Consistency Check Reoccurrence = 168 hrs
Next Consistency check launch = 10/10/2015, 03:00:00
Patrol Read Reoccurrence = 168 hrs
Next Patrol Read launch = 10/10/2015, 03:00:00
Battery learn Reoccurrence = 672 hrs
Next Battery Learn = 10/30/2015, 18:00:00
OEMID = LSI
Drive Groups = 1
TOPOLOGY :
========
-------------------------------------------------------------------------
DG Arr Row EID:Slot DID Type State BT Size PDC PI SED DS3 FSpace
-------------------------------------------------------------------------
0 - - - - RAID6 Optl N 54.569 TB dsbl N dflt N
0 0 - - - RAID6 Optl N 54.569 TB dsbl N N dflt N
0 0 0 0:1 6 DRIVE Onln N 5.456 TB dsbl N N dflt -
0 0 1 0:2 7 DRIVE Onln N 5.456 TB dsbl N N dflt -
0 0 2 0:3 2 DRIVE Onln N 5.456 TB dsbl N N dflt -
0 0 3 0:4 8 DRIVE Onln N 5.456 TB dsbl N N dflt -
0 0 4 0:5 9 DRIVE Onln N 5.456 TB dsbl N N dflt -
0 0 5 0:6 11 DRIVE Onln N 5.456 TB dsbl N N dflt -
0 0 6 0:7 3 DRIVE Onln N 5.456 TB dsbl N N dflt -
0 0 7 0:8 4 DRIVE Onln N 5.456 TB dsbl N N dflt -
0 0 8 0:9 10 DRIVE Onln N 5.456 TB dsbl N N dflt -
0 0 9 0:10 13 DRIVE Onln N 5.456 TB dsbl N N dflt -
0 0 10 0:11 5 DRIVE Onln N 5.456 TB dsbl N N dflt -
0 0 11 0:12 12 DRIVE Onln N 5.456 TB dsbl N N dflt -
-------------------------------------------------------------------------
© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 20 of 25
DG=Disk Group Index|Arr=Array Index|Row=Row Index|EID=Enclosure Device ID
DID=Device ID|Type=Drive Type|Onln=Online|Rbld=Rebuild|Dgrd=Degraded
Pdgd=Partially degraded|Offln=Offline|BT=Background Task Active
PDC=PD Cache|PI=Protection Info|SED=Self Encrypting Drive|Frgn=Foreign
DS3=Dimmer Switch 3|dflt=Default|Msng=Missing|FSpace=Free Space Present
Virtual Drives = 1
VD LIST :
=======
----------------------------------------------------------
DG/VD TYPE State Access Consist Cache sCC Size Name
----------------------------------------------------------
0/0 RAID6 Optl RW Yes RAWBD - 54.569 TB
----------------------------------------------------------
Cac=CacheCade|Rec=Recovery|OfLn=OffLine|Pdgd=Partially Degraded|dgrd=Degraded
Optl=Optimal|RO=Read Only|RW=Read Write|HD=Hidden|B=Blocked|Consist=Consistent|
R=Read Ahead Always|NR=No Read Ahead|WB=WriteBack|
AWB=Always WriteBack|WT=WriteThrough|C=Cached IO|D=Direct IO|sCC=Scheduled
Check Consistency
Physical Drives = 12
PD LIST :
=======
-----------------------------------------------------------------------
EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp
-----------------------------------------------------------------------
0:1 6 Onln 0 5.456 TB SAS HDD N N 4 KB ST6000NM0014 U
0:2 7 Onln 0 5.456 TB SAS HDD N N 4 KB ST6000NM0014 U
0:3 2 Onln 0 5.456 TB SAS HDD N N 4 KB ST6000NM0014 U
0:4 8 Onln 0 5.456 TB SAS HDD N N 4 KB ST6000NM0014 U
0:5 9 Onln 0 5.456 TB SAS HDD N N 4 KB ST6000NM0014 U
0:6 11 Onln 0 5.456 TB SAS HDD N N 4 KB ST6000NM0014 U
0:7 3 Onln 0 5.456 TB SAS HDD N N 4 KB ST6000NM0014 U
0:8 4 Onln 0 5.456 TB SAS HDD N N 4 KB ST6000NM0014 U
0:9 10 Onln 0 5.456 TB SAS HDD N N 4 KB ST6000NM0014 U
0:10 13 Onln 0 5.456 TB SAS HDD N N 4 KB ST6000NM0014 U
0:11 5 Onln 0 5.456 TB SAS HDD N N 4 KB ST6000NM0014 U
0:12 12 Onln 0 5.456 TB SAS HDD N N 4 KB ST6000NM0014 U
-----------------------------------------------------------------------
© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 21 of 25
EID-Enclosure Device ID|Slt-Slot No.|DID-Device ID|DG-DriveGroup
DHS-Dedicated Hot Spare|UGood-Unconfigured Good|GHS-Global Hotspare
UBad-Unconfigured Bad|Onln-Online|Offln-Offline|Intf-Interface
Med-Media Type|SED-Self Encryptive Drive|PI-Protection Info
SeSz-Sector Size|Sp-Spun|U-Up|D-Down|T-Transition|F-Foreign
UGUnsp-Unsupported|UGShld-UnConfigured shielded|HSPShld-Hotspare shielded
CFShld-Configured shielded|Cpybck-CopyBack|CBShld-Copyback Shielded
Cachevault_Info :
===============
------------------------------------
Model State Temp Mode MfgDate
------------------------------------
CVPM03 Optimal 31C - 2015/01/06
------------------------------------
[root@gluster5 ~]# /opt/MegaRAID/storcli/storcli64 /c0 show
Generating detailed summary of the adapter, it may take a while to complete.
Controller = 0
Status = Success
Description = None
Product Name = Cisco 12G SAS Modular Raid Controller
Serial Number = SV50239847
SAS Address = 578da6e715b66710
PCI Address = 00:0a:00:00
System Time = 10/07/2015 08:38:50
Mfg. Date = 01/09/15
Controller Time = 10/07/2015 15:38:48
FW Package Build = 24.5.0-0020
BIOS Version = 6.19.05.0_4.16.08.00_0x06080500
FW Version = 4.250.00-3632
Driver Name = megaraid_sas
Driver Version = 06.803.01.00-rh1
Vendor Id = 0x1000
Device Id = 0x5D
SubVendor Id = 0x1137
SubDevice Id = 0xDB
Host Interface = PCIE
Device Interface = SAS-12G
Bus Number = 10
© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 22 of 25
Device Number = 0
Function Number = 0
Drive Groups = 1
TOPOLOGY :
========
-------------------------------------------------------------------------
DG Arr Row EID:Slot DID Type State BT Size PDC PI SED DS3 FSpace
-------------------------------------------------------------------------
0 - - - - RAID6 Optl N 54.569 TB dsbl N N dflt N
0 0 - - - RAID6 Optl N 54.569 TB dsbl N N dflt N
0 0 0 0:1 6 DRIVE Onln N 5.456 TB dsbl N N dflt -
0 0 1 0:2 7 DRIVE Onln N 5.456 TB dsbl N N dflt -
0 0 2 0:3 2 DRIVE Onln N 5.456 TB dsbl N N dflt -
0 0 3 0:4 8 DRIVE Onln N 5.456 TB dsbl N N dflt -
0 0 4 0:5 9 DRIVE Onln N 5.456 TB dsbl N N dflt -
0 0 5 0:6 11 DRIVE Onln N 5.456 TB dsbl N N dflt -
0 0 6 0:7 3 DRIVE Onln N 5.456 TB dsbl N N dflt -
0 0 7 0:8 4 DRIVE Onln N 5.456 TB dsbl N N dflt -
0 0 8 0:9 10 DRIVE Onln N 5.456 TB dsbl N N dflt -
0 0 9 0:10 13 DRIVE Onln N 5.456 TB dsbl N N dflt -
0 0 10 0:11 5 DRIVE Onln N 5.456 TB dsbl N N dflt -
0 0 11 0:12 12 DRIVE Onln N 5.456 TB dsbl N N dflt -
-------------------------------------------------------------------------
DG=Disk Group Index|Arr=Array Index|Row=Row Index|EID=Enclosure Device ID
DID=Device ID|Type=Drive Type|Onln=Online|Rbld=Rebuild|Dgrd=Degraded
Pdgd=Partially degraded|Offln=Offline|BT=Background Task Active
PDC=PD Cache|PI=Protection Info|SED=Self Encrypting Drive|Frgn=Foreign
DS3=Dimmer Switch 3|dflt=Default|Msng=Missing|FSpace=Free Space Present
Virtual Drives = 1
VD LIST :
=======
----------------------------------------------------------
DG/VD TYPE State Access Consist Cache sCC Size Name
----------------------------------------------------------
0/0 RAID6 Optl RW Yes RAWBD - 54.569 TB
----------------------------------------------------------
Cac=CacheCade|Rec=Recovery|OfLn=OffLine|Pdgd=Partially Degraded|dgrd=Degraded
Optl=Optimal|RO=Read Only|RW=Read Write|HD=Hidden|B=Blocked|Consist=Consistent|
© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 23 of 25
R=Read Ahead Always|NR=No Read Ahead|WB=WriteBack|
AWB=Always WriteBack|WT=WriteThrough|C=Cached IO|D=Direct IO|sCC=Scheduled
Check Consistency
Physical Drives = 12
PD LIST :
=======
-----------------------------------------------------------------------
EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp
-----------------------------------------------------------------------
0:1 6 Onln 0 5.456 TB SAS HDD N N 4 KB ST6000NM0014 U
0:2 7 Onln 0 5.456 TB SAS HDD N N 4 KB ST6000NM0014 U
0:3 2 Onln 0 5.456 TB SAS HDD N N 4 KB ST6000NM0014 U
0:4 8 Onln 0 5.456 TB SAS HDD N N 4 KB ST6000NM0014 U
0:5 9 Onln 0 5.456 TB SAS HDD N N 4 KB ST6000NM0014 U
0:6 11 Onln 0 5.456 TB SAS HDD N N 4 KB ST6000NM0014 U
0:7 3 Onln 0 5.456 TB SAS HDD N N 4 KB ST6000NM0014 U
0:8 4 Onln 0 5.456 TB SAS HDD N N 4 KB ST6000NM0014 U
0:9 10 Onln 0 5.456 TB SAS HDD N N 4 KB ST6000NM0014 U
0:10 13 Onln 0 5.456 TB SAS HDD N N 4 KB ST6000NM0014 U
0:11 5 Onln 0 5.456 TB SAS HDD N N 4 KB ST6000NM0014 U
0:12 12 Onln 0 5.456 TB SAS HDD N N 4 KB ST6000NM0014 U
-----------------------------------------------------------------------
EID-Enclosure Device ID|Slt-Slot No.|DID-Device ID|DG-DriveGroup
DHS-Dedicated Hot Spare|UGood-Unconfigured Good|GHS-Global Hotspare
UBad-Unconfigured Bad|Onln-Online|Offln-Offline|Intf-Interface
Med-Media Type|SED-Self Encryptive Drive|PI-Protection Info
SeSz-Sector Size|Sp-Spun|U-Up|D-Down|T-Transition|F-Foreign
UGUnsp-Unsupported|UGShld-UnConfigured shielded|HSPShld-Hotspare shielded
CFShld-Configured shielded|Cpybck-CopyBack|CBShld-Copyback Shielded
Cachevault_Info :
===============
------------------------------------
Model State Temp Mode MfgDate
------------------------------------
CVPM03 Optimal 31C - 2015/01/06
------------------------------------
© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 24 of 25
Appendix B: Gluster Volume-Creation Script
[root@gluster5 images]# cat gluster-volume-create-c240
gluster volume create gv0 replica 2 gluster5:/rhs/brick1/gv0
gluster6:/rhs/brick1/gv0 gluster7:/rhs/brick1/gv0 gluster8:/rhs/brick1/gv0
Appendix C: FIO Profile Used on Each Client Driver for the Random Test
WriteRand.FIO
[global]
numjobs=2
iodepth=1
ioengine=libaio
direct=1
directory=/fio/
group_reporting
fsync_on_close=1
runtime=300
[test]
name=randwrite
rw=randwrite
bs=[4k|8k|16k|32k]
size=512g
ReadRand.FIO
[global]
numjobs=2
iodepth=1
ioengine=libaio
direct=1
directory=/fio/
group_reporting
runtime=300
[test]
name=randread
rw=randread
bs=[4k|8k|16k|32k]
size=512g
© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 25 of 25
About Red Hat
Red Hat is the world's leading provider of open source software solutions, using a community-powered approach to
reliable and high-performing cloud, Linux, middleware, storage, and virtualization technologies. Red Hat also offers
award-winning support, training, and consulting services. As a connective hub in a global network of enterprises,
partners, and open source communities, Red Hat helps create relevant, innovative technologies that liberate
resources for growth and prepare customers for the future of IT.
Printed in USA C11-736051-00 10/15