NVMe over Fabrics

NVMe over FabricsBuilding blocks for the next generation of SAN connectivity

October 2021

© Copyright 2020 Dell Inc.2 of 32Restricted - Confidential

Agenda

• What are NVMe™ and NVMe over Fabrics™?

• Workloads and use cases by Fabric type

• What do customers expect from NVMe over Fabrics?

• Project Gilgit


0 2 4 6 8 10 12

HDD

ms

HDD vs SSD

Read Time Transfer Time Misc SSD Link Transfer Platform & Adapter Software

Source: Intel Measurements

Source: Jim Handy, SDC 2015

https://www.snia.org/sites/default/files/SDC15_presentations/gen_sessions/JimHandy_The_Long-Term_Future_PrintVersion.pdf

Time scale in

milliseconds



0 2 4 6 8 10 12

HDD

ms

HDD vs SSD


Read Access timeTiny

Rainbow




Time scale in

milliseconds



0 2 4 6 8 10 12

HDD

SSD

ms

HDD vs SSD





Time scale in

milliseconds



0 20 40 60 80 100 120

HDD

SSD

us

HDD vs SSD





Time scale in

microseconds



0 20 40 60 80 100 120

HDD

SSD

us

HDD vs SSD


Transfer Time

Misc. SSD

Media Access Time

Platform & Adapter

Software




Link Transfer



0 20 40 60 80 100 120

MLC NAND SATA 3 ONFi 2

us

SSD Access times







0 20 40 60 80 100 120



us

SSD Access times


ONFi 3 reduces transfer time






0 20 40 60 80 100 120



MLC NAND PCIe x4 G3 ONFi 3

us

SSD Access times



PCIe reduces Link Transfer & Platform & Adapter time






0 20 40 60 80 100 120




Future NVM PCIe x4 G3

us

SSD Access times




NVM Shrinks Read, Transfer & Misc SSD Times






0 20 40 60 80 100 120




Future NVM PCIe x4 G3

us

SSD Access times




NVMe to replace the SCSI SW

Reduce latency down to ~<10us






• Non-Volatile Memory Devices (NVM/SSD/Flash drives) can use

a PCIe interface rather than a serial I/O controller.

NVMe™, and NVMe™ over TCP

PCIe

PCIe

NVMe

NVMe over fabrics

PCIe

NVMe

IP

Network

• The Non-Volatile Memory Express (NVMe) Base specification

that controls the PCIe interface of an SSD/Flash (NVM) drive. – 64K queues with up to 64K commands per queue

• The NVMe over Fabrics Transport specifications describe how

NVMe commands may be carried over a fabric.– The NVMe over Fabrics (NVMe-oF) Transports include: FC (NVMe/FC),

RDMA (NVMe/RoCE, NVMe/IB, and NVMe/iWARP) and TCP (NVMe/TCP).

– TCP’s advantage is support for arbitrary networks without specialized network

configuration (e.g., Lossless operation) or the need to specialized adapters.

– New standards TP-8009 and TP-8010 for NVMe/TCP: Dell and other vendors

collaborate to design standardized automated discovery process.

© Copyright 2020 Dell Inc.14 of 32Restricted - Confidential Pre-GA: Material subject to change.

NVMe-oF™ EvolutionFrom SCSI to NVMe

• Application running on a host that is accessing

external array-based storage via either FC or

iSCSI.

Application

SCSI

SCSI

FCPiSCSI NVMe-oF

SCSI

FCPiSCSI

NVMe

SATA-SSD

Ho

st

Sto

rag

e A

rra

y

NVMe-oF

NVMe

NVMe-SSD

SCSI

Local SATA-SSD Local NVMe-SSD

• NVMe Drives were first introduced on the

host in 2015 and were used mainly for

caching and boot drives

• NVMe-SSDs improve storage array

performance locally but using the SCSI

protocol can add significant latency.

• NVMe-oF™ can run over either Ethernet or

Fibre Channel with low latency.

Fibre

Channel

IP

Network


Agenda

• What are NVMe and NVMe over Fabrics?



• Project Gilgit


NVMe/TCPNVMe/RoCE or IB

Workloads and use cases by Fabric

Logos are indicative of workload characteristics only.

High Perf Use Cases

General purpose use cases

BCA = Business Critical Apps

HCI = Hyper Converged Infrastructure

EBOF = Ethernet Bunch of Flash

vMotion

EBOF

FC-NVMe

In-house

BCAs

Storage

vMotion

Business Critical Apps

Specialized Use Cases


Agenda

• What are NVMe and NVMe over Fabrics?



• Project Gilgit


Operational simplicity

Performance Applications

Scale

SecurityCost

23

What do customers expect from

NVMe over Fabrics?


Configuration used for comparison of fabric types

• The test results/data used to generate the following graphs are based on testing performed at Dell

Technologies.

• These results are only intended to compare the fabric types that are available for use.

– In other words, all things being equal on both ends, the results demonstrate the benefit of using one fabric versus

another.

• These test results DO NOT represent what end-users will experience when using one fabric versus

another to access storage capacity on every storage platform.

– This will be much more dependent on the storage platform implementation than on the fabric in use.

• The test configuration consisted of:

1. An EZFIO client running on a physical server running Linux that has an appropriate adapter type installed

2. All adapters used were either 32GFC, 25GbE or 100GbE based.

3. A single 32GFC or 100GbE switch was used for “SAN” connectivity

4. An LIO Target running on a physical server running Linux that is using RAMDISK for storage capacity.

• The Test results shown are all 4k Random Writes performed with varying numbers of threads


Defining Hero Numbers

• Hero numbers are:– performance metrics that are captured at the extreme ends of the support envelop in order to highlight

what is possible.

– usually focus on Latency and IOPS/BW

– typically captured under ideal conditions (e.g., non-oversubscribed)

• Hero numbers do not:– directly correlate to a performance benefit many end-users will experience with their workloads

– capture the infrastructure ecosystem implications of using one protocol versus another. This is not

only about CapEx, OpEx and Security… You need to consider the operating environment as well as

the end-devices/appliances and their support for the protocol.


Typical operating

range for Web, DB,

and file access

applications

Hero numbers

IOPS


Latency

Typical operating range

for Web, DB, and file

access applications

(No Congestion)

Hero numbers


I/Os per Percent CPU Utilization

Typical operating range for

Web, DB,

and file access applications

FC @ 32GFC

TCP & ROCE @

25GbE


Why TCP vs. RoCE?

Aspect RoCE TCP/IP

Performance Excellent Good, getting better

Interoperability Fair, getting better Excellent

Interop Testing costs High (FC-like) Moderate

Network congestion impacts Visible, unexpected Moderate, expected

Network management impacts New protocol, missing

end-to-end functionality

Part of normal

operations

poor excellent

• NVMe-oF/Ethernet provides standard, interoperable, high-speed, light-weight, low-latency, cost effective block storage access

• 25GE provides essentially the same throughput as 32G FC, but at a fraction of the cost

• NVMe-oF/TCP delivers better performance and reduced overhead compared to typical iSCSI

• TCP/IP for NVMe-oF transport just works by default

• Specialized configuration of TCP is not required

• Realistic performance is similar to RoCE (better as NVMe/TCP offloads emerge)

• TCP/IP is a better fit to Edge, IoT, Client deployments due to price & hardware

• TCP/IP allows a wide variety of network topologies (fully routable and fully flow controlled as needed)




Scale

SecurityCost

30


NVMe Fabric type comparison

FC-NVMe NVMe/RoCE NVMe/TCP

High Speed Performance

Software Defined Storage

Centralized Provisioning1

State Change Notifications

Edge/Distributed System at Scale

Cloud Operating Model/Automation

CapEx Cost Advantage

OP

EX

ben

efi

ts

1. Source: https://brasstacksblog.typepad.com/brass-tacks/2017/12/nvme-over-fabrics-discovery-problem.html


Initial Configuration Steps by Fabric type

Single Host to Single Target

Transport Protocol Host Steps Network Steps Storage Steps Total

FC 2 5 7 14

FC-NVMe 2 5 7 14

NVMe/TCP 4->1 3 8 15->12

NVMe/RoCE 5->2 7 9 21->18

Source: http://brasstacksblog.typepad.com/brass-tacks/2012/02/fc-and-fcoe-versus-iscsi-network-centric-versus-end-node-centric-provisioning.html

Source: http://brasstacksblog.typepad.com/brass-tacks/2015/05/fibre-channel-is-better-than-ethernet.html

http://brasstacksblog.typepad.com/brass-tacks/2012/02/fc-and-fcoe-versus-iscsi-network-centric-versus-end-node-centric-provisioning.html

http://brasstacksblog.typepad.com/brass-tacks/2015/05/fibre-channel-is-better-than-ethernet.html




Scale

SecurityCost

33


Backend

RoCE or IB

Network

JBOF

Frontend

Embedded Scale Out Back End


FC SAN Alternative

▪SAN features at higher performance▪ Better utilization:

capacity, rack space, and power

▪ Scalability▪Management▪ Fault isolation

▪Driver support in Linux, VMware and Windows(3rd party)

Ethernet InfiniBand

Fibre Channel


Compute Storage Disaggregation

Switch

Compute

Compute

Compute

Compute

Compute

Compute

Compute

Compute

Compute

Compute

Compute

Switch

Compute

Compute

Compute

Compute

Compute

Compute

Compute

Compute

Compute

▪Also called

Composable

Infrastructure and

Rack Scale

▪NVMe over

Fabrics/RoCE enables

with nearly local disk

performance

▪NVMe/TCP may be

used depending on

performance

requirements


Hyperconverged and Scale-Out Storage

▪Scale-out ▪ Cluster of commodity servers▪ Software provides storage

functions

▪Hyperconverged collapses compute & storage▪ Integrated compute-storage

nodes & software▪NVMe/RoCE performs like

local/direct-attached SSD▪NVMe/TCP may be used

depending on performance requirements

Scale out Storage

Mellanox x86 Switch

Compute Nodes

Storage Application

VM VMVM VM

NVMe NVMe NVMe

NVMe NVMe NVMe

Storage

App

HCI Nodes


Fabrics at Scale

Cross Rack / Large Cluster

NVMe/TCP/RoCE

NVMe/TCP

NVMe/TCP/RoCE/FC

StorageController

JBOF

Compute

NVMe/RoCE/IB

NVMe/TCP/RoCE/FC

StorageController

JBOF

Compute

Intra Rack / Smaller Scale

NVMe/RoCE/IB

StorageArray

GPU

Remote Datacenter(s)

GPU

NVMe/RoCE

NVMe/TCP/RoCE/FC




Scale

SecurityCost

39


Must secure NVMe Data in flight and rest

Potential DC NVMe-oF Security Threats

Sniffing

Storage Traffic

Data

Corruption

Storage

Masquerading

Session Hijacking


Securing NVMe-oF

FC-SP-2 (FC-NVMe)

▪ FC-SP-2 is an ANSI/INCITS standard (2012) that defines protocols to –

▪ Authenticate Fibre Channel entities

▪ Setup session encryption keys

▪ Negotiate parameters to ensure per frame integrity and confidentiality

▪ Define and distribute security policies over FCP and FC-NVMe

▪ Concurrent FCP and FC-NVMe

▪ HBA provide full offload

TLS for NVMe/TCP

• TLS is a protocol suite defined by the IETF –

• Privacy and data integrity between two communicating applications

• Higher-level protocols like NVMe that can layer on top of the TLS protocol transparently.

• NVMe-oF over TCP already supports TLS

• The NVM Express Technical Proposal TP 8011 added support for TLS 1.3

• Many NICs can provide full offload

IPsec for

NVMe/TCP/RoCE

▪ IPsec is a secure network protocol

suite defined by the IETF –

▪ Authenticates and encrypts the

data packets over IP networks

▪ Higher-level protocols like NVMe that

can layer on top of the IPsec protocol

transparently.

▪ NVMe-oF/TCP/RoCE already

supports IPSEC

▪ Many NICs can provide full offload


NVMe/TCP Security

• Authentication– Authentication support for DH-HMAC-CHAP Negotiation (NVMe.org TP 8006) in the NVMe-oF

Transport Layer.

– Including support for a 3rd Party Authentication Verifier (such as RADIUS) workflow to interface with

the NVMe-oF Transport Layer.

• Transport Security– TLS Support (TLS 1.3 via NVMe.org TP8011) in the NVMe-oF Transport Layer.

– Includes TLS control plane.

• Redfish/DMTF Credential Bootstrap– As discussed in Boot; mechanism works for Boot and non-Boot volumes for credential provisioning.

– It’s mandatory for Boot (Security).

– Doorbell (DSP0270 v1.3 ratified, Redfish structure still in definition)


Agenda

• What is NVMe over Fabrics?



• Project Gilgit


Configuration Steps with Direct Discovery (Existing)

Direct

Discovery

Controller

(DDC)

IO

Controller

NVM SubsystemIP Network

NVMe over

TCP Storage

Adapter

1

1

2

Host sends connect to

Discovery Controller at

IP Address supplied by

admin

Storage admin

provisions

Namespaces

(Storage) to the Host

NQN3 Host Admin uses nvme

connect-all to Discover

and connect to IO

Controllers on that

subsystem.

2

Host

3

4 Repeat 1-3 on all

Hosts for each

subsystem


Configuration Steps with Centralized Discovery (New)

Direct

Discovery

Controller

(DDC)

IO

Controller

NVM SubsystemIP Network

NVMe over

TCP Storage

Adapter

1 Zoning performed on

CDC (optional)

3 After zoning, Host

receives AEN, uses get

log page, and connects

to each IO Controller

2

Host

3

4 Repeat 1-2 for each

Hosts on each

subsystem

Centralized

Discovery

Controller

(CDC)

0 Host and subsystems

automatically discover

the CDC, connect to it

and Register

Discovery info

2 Storage admin

provisions

namespaces to the

Host NQN. Storage

may send zoning info

to CDC

1

0

0


Direct vs Centralized Discovery at scale

Direct Discovery config steps

1. Host: Determine subsystem Discovery controller IP -> connect

2. Storage: Provision storage

3. Host: Discover / connect all

Centralized Discovery config steps1. Host: N/A

2. CDC: Configure Zoning (optional)

3. Storage: Provision storage

What the chart doesn’t show1. Direct becomes impractical @ >64 hosts

2. Direct requires interaction with each host every time a storage subsystem is added or removed.

3. Direct may lead to extended discovery time if many subsystem interfaces are present.


Network Fabric

Automated End-to-End Discovery

• Enables end devices to discover

other end devices without manual

configuration

• Avoids iSCSI like user experience

Storage Arrays

Automated Fabric Formation

• Enables End-to-End reachability

• Standards based

Switch Switch

Switch Switch

Switch

Ethernet/IP Fabric formation services

• SFS (Automated Underlay Config) Dell

only

• Many other solutions available

NVMe/TCP Discovery services

• Discovery Service (CDC service)

• Discovery Client (CDC client)

• OS Specific - SW based

• CDC – See TP-8010

• CDC Discovery - See TP-8009

FC Fabric formation services

• Vendor Specific

• Non-interoperable

Host OS

ESX, Linux, etc

Discovery

Client

Discovery

Client

FC Discovery services

• Discovery Service

• Vendor Specific (e.g., Name

Server)

• Discovery Client

• HBA specific – HW based

Discovery: FC vs NVMe/TCP

NVMe/TCP

Discovery

Service

Underlay Fabric

Config Service

Fibre Channel


Introducing the NVMe IP SAN - Powered by Dell TechnologiesModern storage connectivity with a standards-based, end-to-end NVMe/TCP portfolio

Automation of complex storage and networking tasks

and management

Reliability with global support & services

Accelerating the evolution of storage connectivity with a modern, automated and secure NVMe IP SAN ecosystem

Agility with cutting-edge NVMe storage technology



De

ll E

MC

Glo

ba

l S

up

po

rt &

Se

rvic

es

VMware ESXPowerEdge

PowerEdge MXPowerStore

PowerMax PowerFlex

PowerSwitch


and management






De

ll E

MC

Glo

ba

l S

up

po

rt &

Se

rvic

es

VMware ESXPowerEdge


PowerMax PowerFlex

PowerSwitch

SmartFabric Services (SFS)Automated Ethernet Storage networking


and management






De

ll E

MC

Glo

ba

l S

up

po

rt &

Se

rvic

es

VMware ESXPowerEdge


PowerMax PowerFlex

PowerSwitch


SmartFabric Storage Software (SFSS)Policy driven storage and data services


and management





Smart Fabric Storage Software (SFSS)Standards based Centralized Discovery Controller (CDC) for NVMe/TCP hosts and targets


– NVMe/TCP end points dynamically discover the SFSS instance

– Listen and respond to mDNS queries from end points in the fabric

• End Point Registration Service

– NVMe-oF end point – host or subsystem register their information with SFSS

• End Point Query Service

– NVMe/TCP Hosts query SFSS to discover the NVMe/TCP subsystems they can communicate with

– NVMe/TCP subsystems query SFSS to discover the NVMe/TCP hosts that can connect to them

Equivalent of Fibre Channel Name Server Database











• Zone Service

– Soft Zoning – Get Log Page responses only include sub-systems zoned for the querying host

– Hard Zoning – Enforcement in the network with ACLs – (Future integration with SFS)


Equivalent of Fibre Channel Zone Server Database











• Zone Service

– Soft Zoning – Get Log Page responses only include sub-systems zoned for the querying host

– Hard Zoning – Enforcement in the network with ACLs – (Future integration with SFS)

• Asynchronous Notifications

– Asynchronous Event Registration – subscribe to state change notifications from end points

– Asynchronous Event Notifications – send notifications to end points for state changes


Equivalent of Fibre Channel Zone Server Database

Equivalent of Fibre Channel Registered State Change Notification

(RSCN)



De

ll E

MC

Glo

ba

l S

up

po

rt &

Se

rvic

es

VMware vCenter

VMware ESXPowerEdge


PowerMax PowerFlex

PowerSwitch


SmartFabric Storage Software (SFSS)Policy driven storage and data services

Dell EMCCloudIQ

Dell OpenManage

Enterprise

Storage Orchestrators


and management





NVMe IP SAN operations

NVMe IP SAN Fabric

SmartFabric Storage Software

SmartFabric Services

Orchestrator(vCenter)

PowerEdge Servers (with vSphere)(NVMe/TCP Hosts)

PowerSwitch Switches

PowerStore Storage(NVMe/TCP Subsystems)

OpenManage NetworkIntegration (OMNI)

1Automated deployment of the fabric based on defined policies

Th

is s

lid

e h

as a

nim

atio

n



NVMe IP SAN Fabric









Th

is s

lid

e h

as a

nim

atio

n

2Server and storage systems discover SFSS



NVMe IP SAN Fabric









Th

is s

lid

e h

as a

nim

atio

n


3Server and storage systems

register with

SFSS



NVMe IP SAN Fabric









Th

is s

lid

e h

as a

nim

atio

n


4Operator or orchestrator reads the SFSS Name Server database and sets up server/storage zones


register with

SFSS




NVMe IP SAN Fabric








5SFSS notifies servers and storage systems of a zoning change

Th

is s

lid

e h

as a

nim

atio

n




register with

SFSS



NVMe IP SAN Fabric









5SFSS notifies servers and storage systems of a zoning change

Th

is s

lid

e h

as a

nim

atio

n

6Servers connect to storage and start transferring data




register with

SFSS


Configuration examples


R740

R740

R740

R740

R740

R740

R740

R740

R740

R740

R740

R740

R740

R740

R740

R740

Storage

SAN A SAN B

R640

SFSS

SAN A / SAN B

Two primary deployment models

R740

R740

R740

R740

R740

R740

R740

R740

R740

R740

R740

R740

R740

R740

R740

R740

Storage

R640

SFSS

Z-Series

S-Series

Converged network


OOB Mgmt Network OOB Mgmt Network

Dell Technologies SAN Interconnects for NVMe/TCP with external SFSS

PowerEdge MX Converged Compute

PowerStoreStorage

PowerEdgeCompute

PowerMaxStorage

PowerFlexSoftware

Defined Storage

ETHERNET LAN CONNECTIVITY

NVMe/TCP SAN A

NVMe/TCP SAN B

NVMe/TCP Interconnect with

SMARTFABIC SERVICES

NVMe/TCP Interconnect with

SMARTFABRIC SERVICES

SM ARTFABRIC STORAGE SOFTWAREAutomated Storage Services for NVMe/TCP

SM ARTFABRIC STORAGE SOFTWAREAutomated Storage Services for NVMe/TCPOM NI

Centralized Management(VM installed on ESXi/KVM

POWERSWITCH POWERSWITCH


OOB Mgmt Network OOB Mgmt Network

Dell Technologies SAN Interconnects for NVMe/TCP with embedded SFSS


PowerStoreStorage

PowerEdgeCompute

PowerMaxStorage

PowerFlexSoftware

Defined Storage

ETHERNET LAN CONNECTIVITY

NVMe/TCP SAN A

NVMe/TCP SAN B

NVMe/TCP Interconnect

with SFS & SFSS

NVMe/TCP Interconnect

with SFS & SFSS

Network & NVMe/TCP Storage Automation Services running locally on the SAN Interconnects

POWERSWITCH POWERSWITCH

OM NICentralized Management(VM installed on ESXi/KVM


Converged LAN / SAN Fabric Deployment for NVMe/TCP


PowerStoreStorage

PowerEdgeCompute

PowerMaxStorage

PowerFlexSoftware

Defined Storage

Ethernet LAN & SAN Connectivity

Converged Fabric for Application & Storage traffic with Network & NVMe/TCP Storage Automation Services

POWERSWITCH

AUTONOMOUS

FABRIC DEPLOYMENT Plug and Play fabric for Compute

& Storage connectivity with


Fabric Interconnect

AUTOMATED STORAGE

FABRIC SERVICESNVMe-oF TCP End Point Policies

driven by SFSS deployed as an

external Virtual Application

INTEGRATED

FABRIC MANAGEMENT Centralized Management for

LAN and Storage Services with

OMNI / vCenter

CONVERGED

NVMe over Fabrics

Documents

Transcript of NVMe over Fabrics