SN0530937-00 Rev. C 4/17 1

White Paper

FC-NVMeNVMe over Fabrics

Fibre Channel – the most trusted fabric can transport NVMe natively

BACKGROUND AND SUMMARY Ever since IBM shipped the world’s first hard disk drive (HDD), the RAMAC 305 in 1956, persistent storage technology has steadily evolved. In the early 1990s, various manufacturers introduced storage devices known today as flash-based or dynamic random access memory (DRAM)-based solid state disk (SSD). The SSDs had no moving (mechanical) components, which allowed them to deliver lower latency and significantly faster access times.

For the enterprise market, while SSDs have made great strides in boosting performance, its interface—the 6Gb per second (Gbp/s) SAS 3 bus—began to hinder further advances in performance. Storage devices then moved to the PCI Express (PCIe) bus, which is capable of up to 500 MB/s per lane for PCIe 2.0 and up to 1000 MB/s per lane for PCIe 3.0. In addition to improved bandwidth, latency is reduced by several microseconds due to a faster interface as well as the ability to directly attach to the chip set or CPU. Today, PCIe SSDs are widely available from an array of manufacturers.

While PCIe SSDs removed the hardware bottleneck of using the SATA interface, these devices continued to use the Advanced Host Controller Interface (AHCI) protocol/command set, which dates back to 2004 and was designed with rotating hard drives in mind. AHCI was created for HDDs, addressing the need for multiple commands to read the data. SSDs do not have this need. Because the first PCIe SSDs used the AHCI command set, they were burdened with the overhead that comes with AHCI. The industry had to develop an interface that eliminated the limits imposed by AHCI.

It wasn’t only the overheads of AHCI that challenged PCIe SSD adoption, since each SSD vendor provided a unique driver for each operating system (OS), with a varying subset of features, creating complexity for customers looking for a homogeneous high speed flash solution for enterprise data centers.

To enable an optimized command set and usher in faster adoption and interoperability of PCIe SSDs, industry leaders have defined the Non Volatile Memory Express (NVMe) standard. NVMe defines an optimized command set that is scalable for the future and avoids burdening the device with legacy support requirements. It also enables standard drivers to be written for each OS and enables interoperability between implementations, reducing complexity and simplifying management.

While HDDs and SSDs evolved, so did Fibre Channel (FC). FC is a high-speed network technology (8Gbps and 16Gbps being the dominant speeds today and a strong roadmap to 32Gbps and beyond) primarily used to connect enterprise servers to HDD- or SSD-based data storage. Fibre Channel is standardized in the T11 Technical Committee of the International Committee for Information Technology Standards (INCITS) and has remained the dominant protocol to access shared storage for over a decade. Fibre Channel Protocol (FCP) is a transport protocol that predominantly transports SCSI commands over Fibre Channel networks. With the advent of NVMe, FC is transforming to natively transport NVMe and this new technology is called FC-NVMe.

SN0530937-00 Rev. C 4/17 2

White PaperFC-NVMe

SCSI AND NVME DIFFERENCES While the SCSI/AHCI interface comes with the benefit of wide software compatibility, it cannot deliver optimal performance when used with SSDs connected via the PCIe bus. As a logical interface, AHCI was developed when the purpose was to connect the CPU/memory subsystem with a much slower storage subsystem based on rotating magnetic media. As a result, AHCI introduces certain inefficiencies when used with SSD devices, which behave much more like DRAM than like spinning media.

The NVMe device interface has been designed from the ground up, capitalizing on the low latency and parallelism of PCIe SSDs, and complementing the parallelism of contemporary CPUs, platforms and

applications. At a high level, the basic advantages of NVMe over AHCI relate to its ability to exploit parallelism in host hardware and software, manifested by the differences in command queue depths, efficiency of interrupt processing, and the number of un-cacheable register accesses, resulting in significant performance improvements across a variety of dimensions.

Table 1 below summarizes high-level differences between the NVMe and AHCI logical device interfaces. Figure 2 shows how the Linux storage stack is simplified when using NVMe. All of these purpose-built features bring out the most efficient access method for interacting with NVMe devices.

Figure 1. Evolution of Disk and Fibre Channel Fabric

Table 1. Efficiently and Parallelism Related Feature Comparison Between SCSI and NVMe

Features Legacy Interface (AHCI) NVMe

Maximum command queues 1 65536

Maximum queue depth 32 commands per queue 65536 commands per queue

Un-cacheable register accesses (2000 cycles each)

4 per command Register Accesses are cacheable

MSI-X A single interrupt 2048 MSI-X interrupts

Interrupt steering No steering Interrupt steering

Efficiency for 4KB commands Command parameters require two serialized host DRAM fetches

Gets command parameters in one 64-byte fetch

Parallelism and multiple threads Requires synchronization lock to issue a command Does not require synchronization

SN0530937-00 Rev. C 4/17 3

White PaperFC-NVMe

Figure 2. A Linux Storage Stack Comparison Between SCSI and NVMe

NVME DEEP DIVE NVMe is a standardized high-performance host controller interface for PCIe storage, such as PCIe SSDs. The interface is defined in a scalable fashion such that it can support the needs of enterprise and client applications in a flexible way. NVMe has been developed by an industry consortium, the NVM Express Workgroup. Version 1.0 of the interface specification was released on March 1, 2011 and version 1.2 was released on November 12, 2014. Today, over 100 companies participate in defining the interface.

• Architected from the ground up for this and next generation non-volatile memory to address enterprise and client system needs

• Developed by an open industry consortium, directed by a 13 company Promoter Group

• Architected for on-motherboard PCIe connectivity

• Designed to capitalize on multi-channel memory access with scalable port width and scalable link speed

The interface has the following key attributes:

• Does not require un-cacheable register reads in the command issue or completion path.

• A maximum of one Memory-Mapped I/O (MMIO) register write is necessary in the command issue path.

• Support for up to 64K I/O queues, with each I/O queue supporting up to 64K commands.

• Priority associated with each I/O queue with a well-defined arbitration mechanism.

• All information to complete a 4KB read request is included in the 64B command itself, ensuring efficient small random I/O operation.

• Efficient and streamlined command set.

• Support for Message Signal Interrupts (MSI), MSI Extended (MSI-X) and interrupt aggregation.

• Support for multiple namespaces.

• Efficient support for I/O virtualization architectures like Single Root I/O Virtualization (SR-IOV).

• Robust error reporting and management capabilities.

As a result of the simplicity, parallelism and efficiently of NVMe, it delivers significant performance gains over Serial Attached SCSI (SAS). Some metrics include:

• For 100% random reads, NVMe has 3x better IOPS than 12Gbps SAS1

• For 70% random reads, NVMe has 2x better IOPS than 12Gbps SAS1

• For 100% random writes, NVMe has 1.5x better IOPs than 12Gbps SAS1

• For 100% sequential reads: NVMe has 2x higher throughput than 12Gbps SAS1

• For 100% sequential writes: NVMe has 2.5x higher throughput than 12Gbps SAS1

In addition to IOPS and throughput, the efficiencies of the command structure described above also reduces CPU cycles by half, as well as reduce latency by more than 200 microseconds than 12 Gbps SAS.

NVME AND FIBRE CHANNEL (FC-NVME) Fibre Channel, more specifically [Fibre Channel Protocol (FCP)] has been the dominant protocol used to connect servers with remote shared storage comprising of HDDs and SSDs. FCP transports SCSI commands encapsulated into the Fibre Channel frame and is one of most reliable and trusted networks in the data center for accessing SCSI-based storage. While FCP can be used to access remote shared NVMe-based storage, such a mechanism requires the interpretation and translation of the SCSI commands encapsulated and transported by FCP into NVMe commands that can be processed by the NVMe storage array. This translation and interpretation can impose performance penalties when accessing NVMe storage and in turn negates the benefits of efficiency and simplicity of NVMe.

1 Source: The Performance Impact of NVMe and NVMe over Fabrics

SN0530937-00 Rev. C 4/17 4

White PaperFC-NVMe

FC-NVMe extends the simplicity, efficiency and end-to-end NVMe model where NVMe commands and structures are transferred end-to-end, requiring no translations. Fibre Channel’s inherent multi-queue capability, parallelism, deep queues, and battle-hardened reliability make it an ideal transport for NVMe across the fabric. FC-NVMe implementations will be backward compatible with Fibre Channel Protocol (FCP) transporting SCSI so a single FC-NVMe adapter will support both SCSI-based HDDs and SSDs, as well as NVMe-based PCIe SSDs.

Figure 3. Current Method of Interacting with NVMe Storage via Fibre Channel (FC)

Figure 4. FC-NVMe Proposes a Native Transport of NVMe with Full Backward Compatibility

QLOGIC AND FC-NVME QLogic is a global leader and technology innovator in high-performance server and storage networking connectivity products and leads the Fibre Channel adapter market having shipped over 17 million ports to customers worldwide. As next-generation data-intensive workloads transition to utilizing low latency NVMe flash-based storage to meet ever increasing user demands, QLogic is combining the lossless, highly deterministic nature of Fibre Channel with NVMe. FC-NVMe targets the performance, application response time, and scalability needed for next generation data centers, while leveraging existing Fibre Channel infrastructures. QLogic is pioneering this effort with industry leaders, which in time, will yield significant operational benefits to data center operators and IT managers.

Corporate Headquarters Cavium, Inc. 2315 N. First Street San Jose, CA 95131 408-943-7100

International Offices UK | Ireland | Germany | France | India | Japan | China | Hong Kong | Singapore | Taiwan | Israel

Follow us:

SN0530937-00 Rev. C 4/17 5

Copyright © 2015 - 2017 Cavium, Inc. All rights reserved worldwide. QLogic Corporation is a wholly owned subsidiary of Cavium, Inc. QLogic and QConvergeConsole are registered trademarks or trademarks of Cavium, Inc. All other brand and product names are registered trade-marks or trademarks of their respective owners.

This document is provided for informational purposes only and may contain errors. Cavium reserves the right, without notice, to make changes to this document or in product design or specifications. Cavium disclaims any warranty of any kind, expressed or implied, and does not guarantee that any results or performance described in the document will be achieved by you. All statements regarding Cavium’s future direction and intent are subject to change or withdrawal without notice and represent goals and objectives only.

White PaperFC-NVMe

Figure 5. QLogic Leads the Charter on FC-NVMe

On Dec. 8, 2015 at the Gartner Data Center, Infrastructure & Operations Management Conference, QLogic demonstrated the industry’s first FC-NVMe solution in conjunction with Brocade Gen 5 Fibre Channel fabric. The QLogic FC-NVMe technology demonstration was aimed at providing a foundation for lower latency and increased performance, while providing improved fabric integration for flash-based storage.

QLogic, Brocade, and other industry leaders are collaborating to drive the standards and expect the Fabrics (FC) Specification 1.0 covering FC-NVMe to be a standard by mid-2016.

ABOUT CAVIUM Cavium, Inc. (NASDAQ: CAVM), offers a broad portfolio of infrastructure solutions for compute, security, storage, switching, connectivity and baseband processing. Cavium’s highly integrated multi-core SoC products deliver software compatible solutions across low to high performance points enabling secure and intelligent functionality in Enterprise, Data Center and Service Provider Equipment. Cavium processors and solutions are supported by an extensive ecosystem of operating systems, tools, application stacks, hardware reference designs and other products. Cavium is headquartered in San Jose, CA with design centers in California, Massachusetts, India, Israel, China and Taiwan.