Cisco UCS S3260 Storage Server and Red Hat Ceph Storage ... · Ceph Storage Performance This...

White Paper

© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. of 27

Cisco UCS S3260 Storage Server and Red Hat Ceph Storage Performance

This document provides a performance analysis of Red Hat Ceph Storage using Cisco

UCS® S3260 Storage Servers, Cisco UCS C220 M4 Rack Servers, Cisco UCS 6300

Series Fabric Interconnects, and Cisco UCS Manager.

April 2017


Contents Executive Summary

Main Findings

Introduction

Technology Overview Cisco Unified Computing System Cisco UCS S3260 Storage Server Cisco UCS C220 M4 Rack Server Cisco UCS Virtual Interface Card 1387 Cisco UCS 6300 Series Fabric Interconnect Cisco Nexus 9332PQ Switch Cisco UCS Manager Red Hat Enterprise Linux 7.3 Red Hat Ceph Storage

Solution Design for Tested Configuration Workload Characterization for Red Hat Ceph Storage Design Principles of Red Hat Ceph Storage on Cisco UCS Solution Overview

Tested Solution Physical Setup Red Hat Enterprise Linux and Ceph Performance Setup Hardware Versions

Benchmark Results Performance Baseline Ceph Benchmarking with the Ceph Benchmark Tool CBT Benchmark Results

Sequential Write Performance Sequential Read Performance Random Read Performance

Summary of Benchmark Results

Recommendations for Cisco UCS S-Series and Red Hat Ceph Storage

Conclusion

For More Information


Executive Summary

This document describes Red Hat Ceph Storage performance using a Cisco UCS® S3260 Storage Server solution.

It is based on a Cisco® Validated Design, Cisco UCS S3260 Storage Server with Red Hat Ceph Storage, and

discusses performance for a Ceph-specific workload. The goal of the document is to show the read and write

performance of Red Hat Ceph Storage for a specific workload based on Cisco Unified Computing System™ (Cisco

UCS) architecture and to provide general Cisco UCS configuration recommendations for Ceph-specific workloads.

Main Findings

The main findings of the tests reported in the document are listed here:

● The write performance of the tested solution is up to 30 MBps per Ceph object-storage daemon (OSD)

node.

● The read performance of the tested solution is up to 95 MBps per Ceph OSD node.

● The Cisco UCS S3260 Storage Server can be used for all types of Red Hat Ceph Storage target workloads.

Introduction

With the continuing development of new technologies and the corresponding growth in the amount of data,

organizations are looking for ways to shift large amounts of unstructured data to more cost-effective and flexible

solutions. However, the challenges resulting from the shift to new technologies, including management, flexibility,

performance, and data protection challenges, make implementing solutions difficult. But failure to implement new

technologies contributes significantly to increased demands on capacity and, ultimately, to infrastructure costs.

Storage now consumes a large amount of an organization's IT hardware budget, and business managers are

beginning to question whether the return is worth the investment.

In seeking solutions, customers have learned that new technologies such as software-defined storage can solve

many problems without reducing availability or reliability. In fact, software-defined storage solutions can deliver

greater availability and reliability, making data centers more enterprise ready than before.

One notable software-defined storage solution is Red Hat Ceph Storage. Ceph Storage is an open, cost-effective,

software-defined storage solution that supports massively scalable cloud and object-storage workloads. It can also

deliver enterprise features and high performance for transaction-intensive workloads, which are predominant for

traditional storage and flash arrays.

Together with Cisco UCS, Ceph Storage can deliver a fully enterprise-ready solution that can manage different

workloads and still remain flexible. The Cisco UCS S3260 Storage Server is an excellent platform to use with the

main types of Ceph workloads, such as throughput- and capacity-optimized workloads. It is also excellent for

workloads with a large number of I/O operations per second (IOPS).

This document addresses the performance for throughput- and capacity-optimized workloads and provides

recommendations about how to use the Cisco UCS S3260 for Ceph Storage.

http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/ucs_s3260_rhceph21.html


Technology Overview

This section introduces the technologies used in the solution described in this document.

Cisco Unified Computing System

Cisco UCS is a state-of-the-art data center platform that unites computing, network, storage access, and

virtualization resources into a single cohesive system.

The main components of Cisco UCS are described here:

● Computing: The system is based on an entirely new class of computing system that incorporates rack-

mount and blade servers using Intel® Xeon

® processor E5 and E7 CPUs. The Cisco UCS servers offer the

patented Cisco Extended Memory Technology to support applications with large data sets and allow more

virtual machines per server.

● Network: The system is integrated onto a low-latency, lossless, 10- or 40-Gbps unified network fabric. This

network foundation consolidates LANs, SANs, and high-performance computing (HPC) networks, which are

separate networks today. The unified fabric lowers costs by reducing the number of network adapters,

switches, and cables, and by decreasing the power and cooling requirements.

● Virtualization: The system unleashes the full potential of virtualization by enhancing the scalability,

performance, and operational control of virtual environments. Cisco security, policy enforcement, and

diagnostic features are now extended into virtualized environments to better support changing business and

IT requirements.

● Storage access: The system provides consolidated access to both SAN storage and network-attached

storage (NAS) over the unified fabric. By unifying the storage access layer, Cisco UCS can access storage

over Ethernet (with Network File System [NFS] or Small Computer System Interface over IP [iSCSI]), Fibre

Channel, and Fibre Channel over Ethernet (FCoE). This approach provides customers with choice for

storage access and investment protection. In addition, server administrators can preassign storage-access

policies for system connectivity to storage resources, simplifying storage connectivity and management for

increased productivity.

Cisco UCS is designed to deliver:

● Reduced total cost of ownership (TCO) and increased business agility

● Increased IT staff productivity through just-in-time provisioning and mobility support

● A cohesive, integrated system that unifies the technology in the data center

● Industry standards supported by a partner ecosystem of industry leaders

● Unified, embedded management for easy-to-scale infrastructure

Cisco UCS S3260 Storage Server

The Cisco UCS S3260 Storage Server (Figure 1.) is a modular, high-density, high-availability dual-node rack

server well suited for service providers, enterprises, and industry-specific environments. It addresses the need for

dense, cost-effective storage for the ever-growing amounts of data. Designed for a new class of cloud-scale

applications, it is simple to deploy and excellent for big data applications, software-defined storage environments

such as Ceph and other unstructured data repositories, media streaming, and content distribution.


Figure 1. Cisco UCS S3260 Storage Server

Extending the capabilities of the Cisco UCS C3000 platform, the Cisco UCS S3260 helps you achieve the highest

levels of data availability. With dual-node capability that is based on the Intel Xeon processor E5-2600 v4 series, it

offers up to 600 terabytes (TB) of local storage in a compact 4-rack-unit (4RU) form factor. All hard-disk drives

(HDDs) can be asymmetrically split between the dual nodes and are individually hot-swappable. The drives can be

built in an enterprise-class Redundant Array of Independent Disks (RAID) redundant design or used in pass-

through mode.

This high-density rack server easily fits in a standard 32-inch-depth rack, such as the Cisco R42610 Rack.

The Cisco UCS S3260 is deployed as a standalone server in both bare-metal or virtualized environments. Its

modular architecture reduces TCO by allowing you to upgrade individual components over time and as use cases

evolve, without having to replace the entire system.

The Cisco UCS S3260 uses a modular server architecture that, using Cisco’s blade technology expertise, allows

you to upgrade the computing or network nodes in the system without the need to migrate data from one system to

another. It delivers:

● Dual server nodes

● Up to 36 computing cores per server node

● Up to 60 drives, mixing a large form factor (LFF) with up to 28 solid-state disk (SSD) drives plus 2 SSD

SATA boot drives per server node

● Up to 512 GB of memory per server node (1 TB total)

● Support for 12-Gbps serial-attached SCSI (SAS) drives

● A system I/O controller with a Cisco UCS Virtual Interface Card (VIC) 1300 platform embedded chip

supporting dual-port 40-Gbps connectivity

● High reliability, availability, and serviceability (RAS) features with tool-free server nodes, system I/O

controller, easy-to-use latching lid, and hot-swappable and hot-pluggable components


Cisco UCS C220 M4 Rack Server

The Cisco UCS C220 M4 Rack Server (Figure 2.) is the most versatile, general-purpose enterprise infrastructure

and application server in the industry. It is a high-density 2-socket enterprise-class rack server that delivers

industry-leading performance and efficiency for a wide range of enterprise workloads, including virtualization,

collaboration, and bare-metal applications. The Cisco UCS C-Series Rack Servers can be deployed as standalone

servers or as part of Cisco UCS to take advantage of Cisco’s standards-based unified computing innovations that

help reduce customers’ TCO and increase their business agility.

Figure 2. Cisco UCS C220 M4 Rack Server

The enterprise-class Cisco UCS C220 M4 server extends the capabilities of the Cisco UCS portfolio in a 1RU form

factor. It incorporates the Intel Xeon processor E5-2600 v4 and v3 product family, next-generation DDR4 memory,

and 12-Gbps SAS throughput, delivering significant performance and efficiency gains. The Cisco UCS C220 M4

delivers outstanding levels of expandability and performance in a compact 1RU package:

● Up to 24 DDR4 DIMMs for improved performance and lower power consumption

● Up to 8 small form-factor (SFF) drives or up to 4 LFF drives

● Support for a 12-Gbps SAS module RAID controller in a dedicated slot, leaving the remaining two PCI

Express (PCIe) Generation 3.0 slots available for other expansion cards

● A modular LAN-on-motherboard (mLOM) slot that can be used to install a Cisco UCS VIC or third-party

network interface card (NIC) without consuming a PCIe slot

● Two embedded 1 Gigabit Ethernet LAN-on-motherboard (LOM) ports

Cisco UCS Virtual Interface Card 1387

The Cisco UCS VIC 1387 (Figure 3.) is a Cisco innovation. It provides a policy-based, stateless, agile server

infrastructure for your data center. This dual-port Enhanced Quad Small Form-Factor Pluggable (QSFP+) half-

height PCIe mLOM adapter is designed exclusively for Cisco UCS C-Series and S3260 Rack Servers. The card

supports 40 Gigabit Ethernet and FCoE. It incorporates Cisco’s next-generation converged network adapter (CNA)

technology and offers a comprehensive feature set, providing investment protection for future software feature

releases. The card can present more than 256 PCIe standards-compliant interfaces to the host, and these can be

dynamically configured as either NICs or host bus adapters (HBAs). In addition, the VIC supports Cisco Data

Center Virtual Machine Fabric Extender (VM-FEX) technology. This technology extends the Cisco UCS fabric

interconnect ports to virtual machines, simplifying server virtualization deployment.

Figure 3. Cisco UCS Virtual Interface Card 1387


The Cisco UCS VIC 1387 provides the following features and benefits:

● Stateless and agile platform: The personality of the card is determined dynamically at boot time using the

service profile associated with the server. The number, type (NIC or HBA), identity (MAC address and

World Wide Name [WWN]), failover policy, bandwidth, and quality-of-service (QoS) policies of the PCIe

interfaces are all determined using the service profile. The capability to define, create, and use interfaces on

demand provides a stateless and agile server infrastructure.

● Network interface virtualization: Each PCIe interface created on the VIC is associated with an interface

on the Cisco UCS fabric interconnect, providing complete network separation for each virtual cable between

a PCIe device on the VIC and the interface on the fabric interconnect.

Cisco UCS 6300 Series Fabric Interconnect

Cisco UCS 6300 Series Fabric Interconnects are core components of Cisco UCS, providing both network

connectivity and management capabilities for the system (Figure 4.). The Cisco UCS 6300 Series offers line-rate,

low-latency, lossless 10 and 40 Gigabit Ethernet, FCoE, and Fibre Channel functions.

Figure 4. Cisco UCS 6300 Series Fabric Interconnect

The Cisco UCS 6300 Series provides the management and communication backbone for the Cisco UCS B-Series

Blade Servers, 5100 Series Blade Server Chassis, and C-Series Rack Servers managed by Cisco UCS. All servers

attached to the fabric interconnects become part of a single, highly available management domain. In addition, by

supporting unified fabric, the Cisco UCS 6300 Series provides both LAN and SAN connectivity for all servers within

the Cisco UCS domain.

The Cisco UCS 6300 Series uses a cut-through network architecture, supporting deterministic, low-latency, line-

rate 10 and 40 Gigabit Ethernet ports, switching capacity of 2.56 terabits per second (Tbps), and 320 Gbps of

bandwidth per chassis, independent of packet size and enabled services. The product family supports Cisco low-

latency, lossless 10 and 40 Gigabit Ethernet unified network fabric capabilities, which increase the reliability,

efficiency, and scalability of Ethernet networks. The fabric interconnects support multiple traffic classes over a

lossless Ethernet fabric from the server through the fabric interconnects. Significant TCO savings can be achieved

with an FCoE-optimized server design in which NICs, HBAs, cables, and switches are consolidated.

The Cisco UCS 6332 32-Port Fabric Interconnect is a 1RU Gigabit Ethernet and FCoE switch offering up to 2.56

Tbps throughput and up to 32 ports. The switch has 32 fixed 40-Gbps Ethernet and FCoE ports.


Both the Cisco UCS 6332UP 32-Port Fabric Interconnect and the Cisco UCS 6332 16-UP 40-Port Fabric

Interconnect have ports that can be configured for the breakout feature that supports connectivity between 40

Gigabit Ethernet ports and 10 Gigabit Ethernet ports. This feature provides backward compatibility to existing

hardware that supports 10 Gigabit Ethernet. A 40 Gigabit Ethernet port can be used as four 10 Gigabit Ethernet

ports. Using a 40 Gigabit Ethernet SFP, these ports on a Cisco UCS 6300 Series Fabric Interconnect can connect

to another fabric interconnect that has four 10 Gigabit Ethernet SFPs. The breakout feature can be configured on

ports 1 to 12 and ports 15 to 26 on the Cisco UCS 6332UP fabric interconnect. Ports 17 to 34 on the Cisco UCS

6332 16-UP fabric interconnect support the breakout feature.

Cisco Nexus 9332PQ Switch

Cisco Nexus® 9000 Series Switches (Figure 5.) include both modular and fixed-port switches that provide a flexible,

agile, low-cost, application-centric infrastructure.

Figure 5. Cisco Nexus 9332PQ Switch

The Cisco Nexus 9300 platform consists of fixed-port switches designed for top-of-rack (ToR) and middle-of-row

(MoR) deployment in data centers that support enterprise applications, service provider hosting, and cloud

computing environments. They are Layer 2 and 3 nonblocking 10 and 40 Gigabit Ethernet switches with up to

2.56 Tbps of internal bandwidth.

The Cisco Nexus 9332PQ Switch is a 1RU switch that supports 2.56 Tbps of bandwidth and over 720 million

packets per second (mpps) across thirty-two 40-Gbps QSFP+ ports

All Cisco Nexus 9300 platform switches use dual- core 2.5-GHz x86 CPUs with 64-GB SSD drives and 16 GB of

memory for enhanced network performance.

With the Cisco Nexus 9000 Series, organizations can quickly and easily upgrade existing data centers to carry

40 Gigabit Ethernet to the aggregation layer or to the spine (in a leaf-and-spine configuration) through advanced,

cost-effective optics that enable the use of existing 10 Gigabit Ethernet fiber (a pair of multimode fiber [MMF]

strands).

Cisco provides two modes of operation for the Cisco Nexus 9000 Series. Organizations can use Cisco NX-OS

Software to deploy the Cisco Nexus 9000 Series in standard Cisco Nexus switch environments (NX-OS mode).

Organizations also can use a hardware infrastructure that is ready to support Cisco Application Centric

Infrastructure (Cisco ACI™

) to take full advantage of an automated, policy-based, systems management approach

(ACI mode).


Cisco UCS Manager

Cisco UCS Manager (Figure 6.) provides unified, embedded management of all software and hardware

components of Cisco UCS across multiple chassis and rack servers and thousands of virtual machines. It supports

all Cisco UCS product models, including Cisco UCS B-Series Blade Servers, C-Series Rack Servers, and M-Series

Modular Servers and Cisco UCS Mini, as well as the associated storage resources and networks. Cisco UCS

Manager is embedded on a pair of Cisco UCS 6300 or 6200 Series Fabric Interconnects using a clustered, active-

standby configuration for high availability. The manager participates in server provisioning, device discovery,

inventory, configuration, diagnostics, monitoring, fault detection, auditing, and statistics collection.

Figure 6. Cisco UCS Manager

An instance of Cisco UCS Manager with all Cisco UCS components managed by it forms a Cisco UCS domain,

which can include up to 160 servers. In addition to provisioning Cisco UCS resources, this infrastructure

management software provides a model-based foundation for streamlining the day-to-day processes of updating,

monitoring, and managing computing resources, local storage, storage connections, and network connections. By

enabling better automation of processes, Cisco UCS Manager allows IT organizations to achieve greater agility

and scale in their infrastructure operations while reducing complexity and risk. The manager provides flexible role-

and policy-based management using service profiles and templates.

Cisco UCS Manager manages Cisco UCS systems through an intuitive HTML 5 or Java user interface and a

command-line interface (CLI). It can register with Cisco UCS Central Software in a multidomain Cisco UCS

environment, enabling centralized management of distributed systems scaling to thousands of servers. Cisco UCS

Manager can be integrated with Cisco UCS Director to facilitate orchestration and to provide support for converged

infrastructure and infrastructure as a service (IaaS).

The Cisco UCS XML API provides comprehensive access to all Cisco UCS Manager functions. The API provides

Cisco UCS system visibility to higher-level systems management tools from independent software vendors (ISVs)

such as VMware, Microsoft, and Splunk as well as tools from BMC, CA, HP, IBM, and others. ISVs and in-house

developers can use the XML API to enhance the value of the Cisco UCS platform according to their unique

requirements. Cisco UCS PowerTool for Cisco UCS Manager and the Python Software Development Kit (SDK)

help automate and manage configurations within Cisco UCS Manager.


Red Hat Enterprise Linux 7.3

Red Hat Enterprise Linux (RHEL) is a high-performing operating system that has delivered outstanding value to IT

environments for more than a decade. More than 90 percent of Fortune Global 500 companies use Red Hat

products and solutions, including Red Hat Enterprise Linux. As the world’s most trusted IT platform, Red Hat

Enterprise Linux has been deployed in mission-critical applications at global stock exchanges, financial institutions,

leading telcos, and animation studios. It also powers the websites of some of the most recognizable global retail

brands.

Red Hat Enterprise Linux:

● Delivers high performance, reliability, and security

● Is certified by the leading hardware and software vendors

● Scales from workstations, to servers, to mainframes

● Provides a consistent application environment across physical, virtual, and cloud deployments

Designed to help organizations make a seamless transition to emerging data center models that include

virtualization and cloud computing, Red Hat Enterprise Linux includes support for major hardware architectures,

hypervisors, and cloud providers, making deployments across physical and different virtual environments

predictable and secure. Enhanced tools and new capabilities in this release enable administrators to tailor the

application environment to efficiently monitor and manage computing resources and security.

Red Hat Ceph Storage

Red Hat Ceph Storage is an open, cost-effective, software-defined storage solution that enables massively

scalable cloud and object-storage workloads. By unifying object storage, block storage, and file storage in one

platform, Ceph Storage efficiently and automatically manages the petabytes of data needed to run businesses

facing massive data growth. Ceph is a self-healing, self-managing platform with no single point of failure. Ceph

enables a scale-out cloud infrastructure built on industry-standard servers that significantly lowers the cost of

storing enterprise data and helps enterprises manage their exponential data growth in an automated fashion.

For OpenStack environments, Ceph Storage is tightly integrated with OpenStack services, including Nova, Cinder,

Manila, Glance, Keystone, and Swift, and it offers user-guided storage lifecycle management. Ceph Storage was

voted the number-one storage option by OpenStack users. The product’s highly tunable, extensible, and

configurable architecture offers mature interfaces for enterprise block and object storage, making it well suited for

archive, rich media, and cloud infrastructure environments.

Ceph Storage is also well suited for object-storage workloads outside OpenStack because it is proven at web scale

and flexible for demanding applications, and it offers the data protection, reliability, and availability that enterprises

demand. It was designed from the foundation for web-scale object storage. Industry-standard APIs allow seamless

migration of, and integration with, an enterprise’s applications. A Ceph object-storage cluster is accessible using

Amazon Simple Storage Service (S3), Swift, and native API protocols.

Ceph has a lively and active open-source community contributing to its innovation. At Ceph’s core is the Reliable

Autonomic Distributed Object Store (RADOS) service, which stores data by spreading it across multiple industry-

standard servers. Ceph uses Controller Replication Under Scalable Hashing (CRUSH), a uniquely differentiated

data placement algorithm that intelligently distributes the data pseudo-randomly across the cluster for better

performance and data protection. Ceph supports both replication and erasure coding to protect data, and it

provides multisite disaster-recovery options.


Red Hat collaborates with the global open-source Ceph community to develop new Ceph features and then

packages changes into a predictable, stable, enterprise-quality software-defined storage product, which is Red Hat

Ceph Storage. This unique development model combines the advantage of a large development community with

Red Hat’s industry-leading support services to offer new storage capabilities and benefits to enterprises.

Solution Design for Tested Configuration

Red Hat Ceph Storage on Cisco UCS is a solution well suited for running different workloads with software-defined

storage on enterprise-proven technology.

Workload Characterization for Red Hat Ceph Storage

One of the most important design considerations for software-defined storage is characterization of the target

workload of your application. With the need for enhanced support for new object-storage workloads over the past

several years, Red Hat Ceph Storage is particularly attractive, but it requires a solid understanding and planning.

The rise of flash-memory storage over the past decade and the ongoing development of software-defined storage

systems have both created new possibilities for optimizing workloads.

One of the main benefits of Ceph Storage is the capability to work on different workloads depending on the needs

of the customer. It can be used and easily optimized on Cisco UCS for various workloads through the flexible use

of different systems and components. As shown in Figure 7, the typical workloads can be characterized as follows:

● IOPS optimized: Sled chassis with a high amount of flash memory for high-IOPS workloads and uses

cases such as MySQL workloads

● Throughput optimized: Standard or dense chassis with a mixture of SSDs and HDDs for use cases such

as rich media

● Capacity optimized: Dense or ultradense chassis with a high number of HDDs for use cases such as

active archives

Figure 7. Workload Profiles for Red Hat Ceph Storage


Among these workloads, throughput- and capacity-optimized workloads present the biggest opportunities for Ceph

Storage. The architectural design for these workloads differs. Throughput-optimized workloads can be reproduced

by adding a small amount of flash memory to the storage layer or by using high computing power in the front-end

servers, whereas capacity-optimized workloads use HDDs only.

A benefit of Ceph Storage on Cisco UCS is that you can mix different target workloads and achieve better usability

and readiness for your data center. For example, your Ceph Storage target workload can include a capacity-

optimized application such as backup and a throughput-optimized web-scale application, both on the same Cisco

UCS S-Series Storage Server.

Overall, Ceph Storage is gaining popularity because it can handle different workloads, allowing IT organizations to

more easily incorporate it.

Design Principles of Red Hat Ceph Storage on Cisco UCS

Based on the workload characterization discussed in the preceding section, the general design of a Red Hat Ceph

Storage solution should consider the following principles, summarized in 0

● The need for scale-out storage: Scalability, dynamic provisioning across a unified name space, and

performance at scale are common needs that people hope to address by adding distributed scale-out

storage to their data centers. For a few use cases, such as primary storage for a scale-up Oracle relational

database management system (RDBMS), traditional storage appliances remain the right solution.

● Target workload: Ceph Storage pools can be deployed to serve three types of workload: IOPS-intensive,

throughput-intensive, and capacity-intensive workloads. As noted in Table 1, server configurations should

be chosen accordingly.

● Storage access method: Ceph Storage supports both block access pools and object access pools within a

single Ceph cluster (additionally, distributed file access is in technical preview at the time of this writing).

Block access is supported on replicated pools. Object access is supported on either replicated or erasure-

coded pools.

● Capacity: Depending on cluster storage capacity needs, standard, dense, or ultradense servers can be

chosen for Ceph storage pools. The Cisco UCS C-Series and S-Series provide several well-suited server

models.

● Fault-domain risk tolerance: Ceph clusters are self-healing following hardware failure. Customers wanting

to reduce the impact on performance and resources during the self-healing process should adhere to the

minimum cluster server recommendations listed in Table 1.

● Data-protection method: With replication and erasure coding, Ceph Storage offers two data-protection

methods that affect the overall design. Erasure-coded pools can provide a better price-to-performance ratio,

and replicated pools typically provide higher absolute performance.


Figure 8. Red Hat Ceph Storage Design Considerations

Table 1presents some technical specifications you should follow for a successful implementation based on the

design principles discussed here.

Table 1. Technical Specifications for Red Hat Ceph Storage

Workload Cluster Size Network CPU and Memory Ratio of OSD Journal to Disk Media

Data Protection

IOPS Minimum of 10 OSD nodes

10 to 40 Gbps 4 to 10 cores per OSD / 16 GB + 2 GB per OSD node

4:1 SSD:NVMe ratio or all NVMe with co-located journals

Ceph RADOS block device (RBD; block) replicated pools

Throughput Minimum of 10 OSD nodes

10 to 40 Gbps (greater than 10 Gbps with more than 12 HDDs or nodes)

1 core per 2 HDDs / 16 GB + 2 GB per OSD node

12-18:1 HDD:NVMe ratio, or 4 or 5:1 HDD:SSD ratio

Ceph RBD (block) replicated pools

Ceph RADOS gateway (RGW; object) replicated pools

Capacity archive

Minimum of 7 OSD nodes

10 Gbps (or 40 Gbps for latency-sensitive workloads)

1 core per 2 HDDs / 16 GB + 2 GB per OSD node

All HDDs with co-located journals

Ceph RGW (object) erasure-coded pools

The tested Ceph Storage on Cisco UCS combination uses a mixed workload of throughput- and capacity-optimized

configurations, as follows:

● Cluster size: 10 OSD nodes

● Network: All Ceph nodes connected with 40 Gbps

● CPU and memory: All nodes configured with 128 GB of memory and more than 28 cores

● OSD disk: Configured for a 6:1 HDD:SSD ratio

● Data protection: Ceph RBD with a replication factor of 3 (3x replication) and erasure coding

● Ceph monitor (MON) nodes: Deployed on Cisco UCS C220 M4 Rack Server

● Ceph OSD nodes: Deployed on Cisco UCS S3260 Storage Server.


Solution Overview

The tested solution using Cisco UCS and Red Hat Ceph Storage is built on a three-part foundation (Figure 9):

● Integration and configuration of the Cisco UCS S3260 Storage Server into Cisco UCS Manager

● Base installation of Red Hat Enterprise Linux and preparation for the next step

● Deployment of Red Hat Ceph Storage

Figure 9. Tested Configuration

Detailed design and deployment steps are presented in the document Cisco UCS S3260 Storage Server with Red

Hat Ceph Storage – Design and Deployment of Red Hat Ceph Storage 2.1 on Cisco UCS S3260 Storage Server.

Tested Solution

The tested solution uses a best-practices configuration. It demonstrates the performance of a mixed SSD and HDD

configuration with a focus on throughput- and capacity-optimized workloads. To get a better understanding of Ceph

performance, the tested solution focused on block storage devices with two different protection methods:

● Replication: Replicated storage pools provide the most common protection method. This protection

method is the only one that Red Hat supports for block devices. The default protection method uses 3x

replication.

● Erasure coding: Erasure-coded pools are useful for cost-effective storage of data such as data in active

archives. This method creates a single copy plus parity and uses n = k + m notation, where k is the number

of data chunks, m is the number of coding chunks, and n is the number of chunks placed by CRUSH over

the OSD nodes. Usually 4 + 2 coding is used, but the values can vary depending on the customer’s

preference. Erasure coding is currently not supported for block devices, but the tested solution gives an

indication of the performance difference between replication and erasure coding.

The following sections describe the physical setup of the tested solution with all the hardware and software

components.




Physical Setup

Figure 10 shows the rack configuration with the following components:

● Two Cisco Nexus 9332PQ Switches for client access

● Two Cisco UCS 6248 fabric interconnects for management of clients

● Two Cisco UCS 6332 fabric interconnects for management of Ceph environment

● Three Cisco UCS C220 M4 servers for Ceph MON

● Five Cisco UCS S3260 servers for Ceph OSD

● Two Cisco UCS chassis with 10 Cisco UCS B200 M3 Blade Servers for Ceph client (24 virtual machines

running Red Hat Enterprise Linux 7.3 configured in all)

Figure 10. Overview of Rack Configuration

Figure 11 shows the network setup for the tested solution. Both Cisco UCS chassis with ten Cisco UCS B200 M3

blades are fully connected with thirty-two 10-Gbps links to both Cisco UCS 6248 fabric interconnects. Each Cisco

UCS 6248 fabric interconnect is connected with twelve 10-Gbps uplinks to the Cisco Nexus 9332PQ, helping

ensure that each Ceph client has a theoretical bandwidth of 20 Gbps or more per blade.

The Ceph environment is fully connected with 40-Gbps network technology, helping ensure that there are no

bandwidth limitations for the OSD nodes. Each Ceph OSD node is connected through two 40-Gbps links to each

Cisco UCS 6332 fabric interconnect. Each Ceph MON node is connected with a single 40-Gbps line to each Cisco

UCS 6332 fabric interconnect. Both Cisco UCS 6332 fabric interconnects are connected with four 40-Gbps links to

each Cisco Nexus 9332PQ.


Figure 11. Network Setup

The tested solution follows the best practice of separating the Ceph public network from the Ceph cluster network.

All Ceph OSD nodes run with one 40-Gbps interface in their own private LANs for the cluster network, leaving all

the network traffic under the Cisco UCS 6332 fabric interconnect. This approach prevents additional network traffic

on the client network. In addition, all Ceph OSD nodes run with one 40-Gbps interface on the public network, the

same as all Ceph MON nodes and all Ceph clients. All network interfaces were running with a maximum

transmission unit (MTU) size of 9000.

Red Hat Enterprise Linux and Ceph Performance Setup

Following the document Cisco UCS S3260 Storage Server with Red Hat Ceph Storage – Design and Deployment

of Red Hat Ceph Storage 2.1 on Cisco UCS S3260 Storage Server, the CPU BIOS settings

(http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/ucs_s3260_rhceph21.html#_Toc474221

977) and the virtual NIC (vNIC) adapter settings

(http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/ucs_s3260_rhceph21.html#_Toc474221

976) were configured for optimal use under Red Hat Enterprise Linux 7.3.

The setup for the OSD and journal drives was as follows:

● OSD HDD



http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/ucs_s3260_rhceph21.html#_Toc474221977)





◦ RAID 0

◦ Access policy: Read Write

◦ Write cache policy: Always Write Back

◦ Drive cache: Disable

◦ Read policy: Normal

◦ I/O policy: Direct

● Journal SSD

◦ JBOD

The configuration of the Ceph Storage cluster follows best practices by Red Hat with modifications in the Ansible

all file and can be consulted at

http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/ucs_s3260_rhceph21.html#_Toc4742220

75.

Hardware Versions

Table 2 provides an overview of the tested hardware configuration.

Table 2. Hardware Versions

Ceph Tier Platform Component Specifics

OSD with 5 Cisco UCS S3260 chassis, each with 2 Cisco UCS C3260 M4 nodes (overall 10 OSD nodes)

Cisco UCS C3260 M4 CPU 2 x Intel Xeon processor E5-2680 v4 CPUs at 2.4 GHz

Memory 8 x 16-GB 2400-MHz DDR4

Network 1 x Cisco UCS VIC 1387 with dual-port 40 Gbps

RAID controller 1 x Broadcom ROC 3316i with 4-GB cache

Storage Boot: 2 x 120-GB SATA Intel DC S3500 SSDs (RAID 1)

Journal: 4 x 400-GB SAS Toshiba PX02SMF040 SSDs (JBOD)

OSD: 24 x 6-TB NL-SAS Seagate ST6000NM0014 drives (RAID 0)

MON with 3 Cisco UCS C220 M4 servers

Cisco UCS C220 M4 CPU 2 x Intel Xeon processor E5-2699 v4 CPUs at 2.2 GHz



RAID controller 1 x Broadcom ROC 3108i

Storage Boot: 2 x 600-GB SAS 10,000-rpm drives (RAID 1)

Client with 10 Cisco UCS B200 M3 servers

Cisco UCS B200 M3 CPU 2 x Intel Xeon processor E5-2680 v2 CPUs at 2.8 GHz



RAID controller 1 x LSI SAS 2004

Storage Boot: 2 x 300-GB SAS 10,000-rpm drives (RAID 1)




Software Distributions and Versions

The required software distribution versions are listed in Table 3.

Table 3. Software Versions

Layer Component Version or Release

Computing (chassis): Cisco UCS S3260 Board controller 1.0.14

Chassis management controller 2.0(13e)

Shared adapter 4.1(2d)

SAS expander 04.08.01.B073

Computing (server node): Cisco UCS C3260 M4

BIOS C3x60M4.2.0.13c

Board controller 2.0

Cisco Integrated Management Controller (IMC) 2.0(13f)

Storage controller 29.00.1-0042

Computing (rack server): Cisco UCS C220 M4S

Adapter 4.1(2d)

BIOS C220M4.2.0.13d

Board controller 32.0

Cisco IMC 2.0(13f)

Cisco FlexFlash controller 1.3.2 build 165

Storage controller 24.12.1-0049

Network: Cisco UCS 6332 fabric interconnect Cisco UCS Manager 3.1(2c)

Kernel 5.0(3)N2(3.12c)

System 5.0(3)N2(3.12c)

Network: Cisco Nexus 9332PQ Switch BIOS 07.59

Cisco NX-OS Software 7.0(3)I5(1)

Software Red Hat Enterprise Linux Server 7.3 (x86_64)

Ceph 10.2.3-13.el7cp

Benchmark Results

Because Ceph is a software-defined storage solution built on various hardware components, it is important to get a

solid understanding of the whole solution by running various performance benchmarks. The recommended

approach is to run specific network and storage base-performance benchmarks at the beginning, before starting a

comprehensive Ceph benchmarking process.

Performance Baseline

To fully understand what the maximum performance of the tested Ceph solution is, the recommended approach is

to perform some simple base testing of network and storage by using Linux tools such as iperf31 and fio

2.

For the base network performance, we conducted various iperf3 tests between all Ceph components. Table 4

provides a summary of the network performance.

1 For more information about iperf3, see https://github.com/esnet/iperf.

2 For more information about fio, see https://github.com/axboe/fio.

https://github.com/esnet/iperf

https://github.com/axboe/fio


Table 4. Network Baseline Performance with iperf3

Ceph OSD Ceph MON Ceph Client

Ceph OSD 39.6 Gbps 39.6 Gbps 9.9 Gbps

Ceph MON 39.5 Gbps 39.7 Gbps 9.9 Gbps

Ceph Client 9.9 Gbps 9.9 Gbps 9.9 Gbps

Additional information about optimizing the maximum network performance on a Cisco UCS S3260 Storage Server

can be found in the document Achieve Optimal Network Throughput on the Cisco UCS S3260 Storage Server.

We next evaluated base performance for the Cisco UCS S3260 HDDs and SSDs. At first, we tested a single HDD

and SSD. Then, to see the maximum performance we could get from a single S3260 node, we ran a performance

baseline test using 24 HDDs and 4 SSDs.

We concentrated on both extremes, running fio tests for a single device and genfio for multiple devices with an

IOdepth of 1 on block sizes of 4 KB and 4 MB. The results are shown in Table 5.

Table 5. SSD and HDD Baseline Performance with fio and genfio

Block Size Fio Workload HDD (RAID 0)

1 Seagate 6-TB HDD

HDD (RAID 0)

24 Seagate 6-TB HDDs

SSD (JBOD)

1 Toshiba 400-GB SSD

SSD (JBOD)

4 Toshiba 400-GB SSDs

4 KB Sequential read 24,900 IOPS 140,000 IOPS 8,400 IOPS 37,400 IOPS

Sequential write 26,880 IOPS 95,100 IOPS 20,400 IOPS 81,300 IOPS

Random read 196 IOPS 2,086 IOPS 10,400 IOPS 30,600 IOPS

Random write 747 IOPS 4,653 IOPS 19,600 IOPS 79,400 IOPS

4 MB Sequential read 226 MBps 5358 MBps 600 MBps 2988 MBpS

Sequential write 156 MBps 3611 MBps 430 MBps 1641 MBps

Random read 167 MBps 2730 MBps 890 MBps 2753 MBps

Random write 185 MBps 2233 MBps 427 MBps 1265 MBps

The performance baseline for the network and disks provided a solid understanding of the maximum performance

that can be achieved on Cisco UCS servers. The results may vary in other configurations; they represent an

average of the components used in the testing reported here.

Ceph Benchmarking with the Ceph Benchmark Tool

The next step in the performance benchmarking process is a comprehensive test of the solution performance using

the tested Ceph cluster.

A common benchmarking tool for examining the performance of a Ceph cluster is the Ceph Benchmark Tool

(CBT)3. CBT is based on Python and can use various benchmark drivers such as radosbench, librbdfio, kvmrbdfio,

and rbdfio. The tests reported here used radosbench, which comes with the ceph-common package. It contains a

benchmarking facility that exercises the cluster using librados, the low-level native object-storage API provided by

Ceph.

3 CBT can be downloaded at https://github.com/ceph/cbt.

http://www.cisco.com/c/en/us/products/collateral/servers-unified-computing/ucs-s-series-storage-servers/Whitepaper_c11-738722.html

https://github.com/ceph/cbt


CBT requires several settings on the administration node and the client nodes, which can be checked at the CBT

website (https://github.com/ceph/cbt). To run the tests, you need to configure different roles for the nodes:

● Administration or head node: This node can access and manage the Ceph cluster and initiates the CBT

command.

● Client nodes: These nodes can access the cluster and generate the load.

● OSD and MON nodes: These nodes have the usual Ceph functions in a Ceph cluster. They collect the

performance data from the CBT benchmark testing and send it to the administration or head node.

The tested solution used a separate Cisco UCS C220 M4 node as the administration or head node. For the CBT

clients, the solution used 10 physical blades with 24 virtual Red Hat Enterprise Linux machines. The CBT OSD and

MON nodes were installed on Cisco UCS S3260 and C220 M4 nodes. Figure 12 shows the CBT setup for the

tested solution.

Figure 12. CBT Setup

https://github.com/ceph/cbt)


An example of a CBT configuration file for initiating the benchmark is shown here. In this setup, 24 CBT clients, 10

OSD nodes, and 3 MON nodes are configured. The benchmark test runs with an object size of 4 MB and a

sequential workload.

cluster:

user: "cephadm"

head: "cephadm"

clients: ["client1", "client2", "client3", "client4", "client5", "client6",

"client7", "client8", "client9", "client10", "client11", "client12", "client13",

"client14", "client15", "client16", "client17", "client18", "client19", "client20",

"client

osds: ["cephosd1", "cephosd2", "cephosd3", "cephosd4", "cephosd5", "cephosd6",

"cephosd7", "cephosd8", "cephosd9", "cephosd10"]

mons:

cephmon1:

a: "192.168.10.111:6789"

cephmon1:

a: "192.168.10.112:6789"

cephmon1:

a: "192.168.10.113:6789"

iterations: 1

rebuild_every_test: False

use_existing: True

clusterid: "ceph"

tmp_dir: "/tmp/cbt"

pool_profiles:

replicated:

pg_size: 4096

pgp_size: 4096

replication: 3

benchmarks:

radosbench:

op_size: [ 4194304 ]

write_only: False

time: 300

concurrent_ops: [ 128 ]

concurrent_procs: 1


use_existing: True

pool_profile: "replicated"

pool_per_proc: False

target_pool: "rados-bench-cbt"

readmode: "seq"

osd_ra: [131072]

For the CBT performance benchmark, all tests were run in sequential order. The tests also used a patch that

helped ensure that the workload of the cluster (CPU and disk) remained under 5 percent because the CBT

benchmark deletes all objects in a pool after each run, which can cause a higher-than-normal load in the cluster.

Note: If you don’t implement the patch, make sure that you wait after each CBT run until the cluster is under

normal load.

CBT Benchmark Results

The CBT benchmarking focused on one single object size of 4 MB, with the Ceph cluster configured with a 6:1

HDD:SSD ratio for throughput- and capacity-optimized workloads. Because latency is not an important factor for

these workloads, the testing concentrated on bandwidth performance only. Testing focused on three benchmarks

for the whole cluster, with a comparison between Ceph replication pools and Ceph erasure-coding pools:

● Sequential write performance

● Sequential read performance

● Random read performance

To provide a better understanding of the result values, the tests used a performance value based on MBps per

OSD node. This value is independent of the size of the cluster and provides a better comparison in the event that

components of the cluster, such as SSDs and HDDs, are changed.

Sequential Write Performance

The sequential write performance test compared the sequential write performance of a replication pool with 3x

replication and the performance of an erasure-coded pool with 4 + 2 coding. Peak performance was 19.7 MBps per

OSD node for the replication pool compared to 30.7 MBps per OSD node for the erasure-coded pool. Erasure-

coded pools benefit from fewer write operations, in contrast to a replication pool with 3x replication, resulting in

higher bandwidth. In general, the write performance depends on the configuration of the journals and the SSD that

is being used. An NVMe SSD, which is also available for the Cisco UCS S3260, would result in higher write

bandwidth. Figure 13 shows the results of the test.


Figure 13. Sequential Write Performance with CBT

Sequential Read Performance

The sequential read performance test compares a 3x replication pool and an erasure-coded 4 + 2 pool. Peak

performance was 82.3 MBps per OSD node for the 3x replication pool and 51.2 MBps per OSD node for the

erasure-coded pool. Replication pools achieve higher bandwidth because fewer read operations are performed: a

single object is read compared to four chunks in an erasure-coded pool. Read operations come straight from the

disks. Figure 14 shows the results of the test.


Figure 14. Sequential Read Performance with CBT

Random Read Performance

The random read performance test compared the random read performance of a 3x replication pool and the

performance of an erasure-coded 4 + 2 pool. The peak bandwidth was about 94.6 MBps per OSD node for the 3x

replication pool and 83.5 MBps per OSD node for the erasure-coded pool. In the tests, more random read

operations were performed than sequential read operations because Read Ahead was not enabled for the read

policy of each OSD node and HDD, and reading is serialized per placement group. Figure 15 shows the results of

the tests.


Figure 15. Random Read Performance with CBT

Summary of Benchmark Results

To summarize the performance benchmarks, the Cisco UCS S3260 Storage Server showed excellent

performance. It also offers additional options to improve the bandwidth. NVMe technology can be added to the

Cisco UCS S-Series servers, increasing performance further. In addition, the S-Series is an excellent platform for

running multiple different Ceph workloads, and it helps customers manage a comprehensive software-defined

storage environment with the easy-to-use Cisco UCS Manager management interface.

Recommendations for Cisco UCS S-Series and Red Hat Ceph Storage

The main advantages of the Cisco UCS S3260 are its flexibility to host a variety of object-storage workloads, its

high network performance, and its management simplicity. The Cisco UCS C220 M4 front-end rack servers can be

considered fixed-configuration servers, and the Cisco USC S3260 servers can be customized in a variety of ways

to meet different Ceph Storage target workloads. Table 6 provides an overview of the recommended configurations

for various workloads.


Table 6. Recommendations for Cisco UCS S3260 with Red Hat Ceph Storage

Workload Component Specifications Comments

IOPS optimized CPU 2 Intel Xeon processor E5-2640 v4 CPUs at 2.4 GHz or higher

Physically separated workload in a dual-node S3260 in which 1 node hosts the IOPS-optimized workload and the other node hosts any other workload

Minimum of 10 dual-node S3260 servers required

Memory 128 GB

Network 10 to 40 Gbps

Storage 1 NVMe for journals and 4 SSDs for OSD nodes

Throughput optimized CPU Up to 2 Intel Xeon processor E5-2695 v4 CPUs CPU and memory can be lower depending on the number of OSD nodes

Minimum of 10 single nodes or 5 dual-node S3260 servers required

Memory 128 to 256 GB

Network 40 Gbps

Storage Up to 3 NVMe or 12 SSDs for journals and up to 48 to 56HDDs for OSD nodes

Capacity optimized CPU Up to 2 Intel Xeon processor E5-2695 v4 CPUs CPU and memory can be lower depending on the number of OSD nodes

Minimum of 7 single nodes or 4 dual-node S3260 servers required

Memory 128 to 256 GB

Network 40 Gbps

Storage Up to 60 HDDs for OSD nodes (optional use of SSDs for journals)

The use of Cisco UCS Manager greatly simplifies management and scaling for the Red Hat Ceph Storage solution

compared to traditional management and scaling operations using the local server BIOS and network switch

management.

Performance can also be enhanced by the use of individual RAID 0 HDD OSD nodes and the enablement of Read

Ahead read policy for each OSD node.

Conclusion

The use of software-defined storage is growing rapidly, and Red Hat Ceph Storage and the Cisco UCS S3260

Storage Server together provide an excellent solution for organizations adopting that technology. The flexibility of

the Cisco UCS S3260 helps customers use Ceph Storage in a variety of ways, reducing overall TCO.

The solution delivers high performance through flexible composable architecture and an easy-to-use management

interface through Cisco UCS Manager, achieving two important features for Ceph Storage. Customers can now

scale independently and achieve high performance for all types of workloads. In benchmark testing, the Cisco UCS

S-Series demonstrated read and write performance that exceeds the current standards and sets a new milestone

for software-defined storage with Ceph Storage.


For More Information

For additional information, see the following:

● Cisco UCS S3260 Storage Server:

http://www.cisco.com/c/en/us/products/servers-unified-computing/ucs-s-series-storage-servers/index.html

● Cisco UCS C220 M4 Rack Server:

http://www.cisco.com/c/en/us/products/servers-unified-computing/ucs-c220-m4-rack-server/index.html

● Cisco UCS 6300 Series Fabric Interconnects:

http://www.cisco.com/c/en/us/products/servers-unified-computing/ucs-6300-series-fabric-

interconnects/index.html

● Cisco UCS Virtual Interface Card 1387:

http://www.cisco.com/c/en/us/products/interfaces-modules/ucs-virtual-interface-card-1387/index.html

● Cisco UCS Manager:

http://www.cisco.com/c/en/us/products/servers-unified-computing/ucs-manager/index.html

● Cisco Validated Design—Cisco UCS S3260 Storage Server with Red Hat Ceph Storage:


● Cisco white paper—Achieve Optimal Network Throughput on the Cisco UCS S3260 Storage Server:

http://www.cisco.com/c/en/us/products/collateral/servers-unified-computing/ucs-s-series-storage-

servers/Whitepaper_c11-738722.html

Printed in USA C11-738915-01 05/17

http://www.cisco.com/c/en/us/products/servers-unified-computing/ucs-s-series-storage-servers/index.html

http://www.cisco.com/c/en/us/products/servers-unified-computing/ucs-c220-m4-rack-server/index.html

http://www.cisco.com/c/en/us/products/servers-unified-computing/ucs-6300-series-fabric-interconnects/index.html

http://www.cisco.com/c/en/us/products/servers-unified-computing/ucs-6300-series-fabric-interconnects/index.html

http://www.cisco.com/c/en/us/products/interfaces-modules/ucs-virtual-interface-card-1387/index.html

http://www.cisco.com/c/en/us/products/servers-unified-computing/ucs-manager/index.html




Cisco UCS S3260 Storage Server and Red Hat Ceph Storage ... · Ceph Storage Performance This...

Documents

Transcript of Cisco UCS S3260 Storage Server and Red Hat Ceph Storage ... · Ceph Storage Performance This...