Paul Luse, Principal Software Engineer

Agenda

• SNIA NVM Programming Model

• PMDK Overview

• Kit Comparison

• PMDK & SPDK Integration

• Potential Use Cases

2

3

NVDIMMs

UserSpace

KernelSpace

Standard

File API

NVDIMM Driver

Application

File System

ApplicationApplication

Standard

Raw Device

Access

Load/Store

Management Library

Management UI

Standard

File API

pmem-AwareFile System

MMU

Mappings

4

• PMDK is a collection of libraries

• Developers pull only what they need

• Low level programming support

• Transaction APIs

• Fully validated

• Performance tuned.

• Open Source & Product neutral

http://pmem.io/

5

Support for

volatilememory

usage

libmemkind

Low level

support for

local persistent

memory

libpmem

Low level support for remote access to persistent memory

librpmem

In Development

Interface to create

arrays of pmem-

resident blocks, of

same size,

atomically updated

Interface for persistent

memory allocation,

transactions and

general facilities

Interface to create

a persistent

memory resident

log file

libpmemblklibpmemlog libpmemobj

C++ C PCJ* Python

* Persistent Collections for Java

6

PMDK SPDK

IO model Synchronous Asynchronous

Requires app changes Yes Sometimes

Data Access Granularity Byte Block

Data Access Style Memory I/O

Media Persistent Memory SSD, PM, Kernel Devices (AIO)

Open Source Yes Yes

7

For example: Let’s say I want to write 4K to my NVM…. (skipping lots of details)

PMDK SPDK

1. Integrate libpmem with application

2. Memory map my PM file on my PM-aware file system using pmem_map_file()

3. Perform assignments directly on the memory addresses

4. Make persistent with pmem_persist()

The memory location represents direct access to the media.

1. Integrate SPDK with application2. Allocate DMA safe memory buffer

and update accordingly.3. Use block device layer API

spdk_bdev_write()

4. Poll for I/O completion

Once the IO has been accepted by the NVMe controller, the memory location is DMA’d by the NVMe controller to the media and a completion is posted to the application.

Intel® Builders 8

PMDK & SPDK independent Integration

• Integrate PMDK w/Application independent of SPDK

• Separate path to faster media via SNIA PM

• Possible Use Cases:

• SPDK path in application already in production, but application has other use cases for high speed persistence,

• Cache

Application

Drivers

<SPDK Defined Block API>

<Driver Specific API>

I/O

PM

by

tes

fs

PMDK

con

tro

l

Block Device Layer

SPDK

Intel® Builders 9

PMDK & SPDK Integration via libpmemblk• Bdev module leveraging libpmemblk to access

persistent memory as blocks

• Libpmemblk addresses block atomicity requirement in software to protect from torn writes

• Bdev API out the top, persistent memory SNIA programming model out the bottom, directly to media.

• Possible Use Cases:

• Stepping stone to PM access via SNIA PM

• Separate path to faster media via block access

• Block cache, block based metadata

Block Device Layer

Application

Drivers

<SPDK Defined Block API>

Libpmemblkbdev module

<Driver Specific API>

I/O

PM

by

tes

Other bdev Modules

Intel® Builders 10

PMDK & SPDK complete Integration• Libpmem integrated into the bdev layer for use by

other bdev functions and bdev modules

• Enables use of PM as persistent storage replacing DRAM in many cases.

• Possible Use Cases:

• Optimal use of SPDK fast path and PM “faster” path

• Data cache

• Metadata storage of any kind

• Write ahead log

Block Device Layer

Application

Drivers

<SPDK Defined Block API>

libpmem

<Driver Specific API>

I/O

PM

by

tes

bbdevmodules

Bdevfunctions

11

Backup

12

• libpmem• Detects types of flush instructions supported by the CPU, and uses best instructions for the platform to

create performance-tuned routines for copying ranges of persistence memory. • libpmemobj is the core library that supports persistent memory allocations, transactions and relies on lower

level libraries for data persistence support.• librpmem:

• Provides low-level support for remote access to persistent memory (pmem) utilizing RDMA-capable NICs. This library can be used to replicate remotely a memory region over RDMA protocol. It utilizes appropriate persistency mechanism based on the remote node’s platform capabilities.

• libpmemlog• support specific usages and are tuned for these usages. For example, libpmemlog is useful where an

application frequently appends to log file. • libpmemblk

• useful to manage application block cache residing in persistent memory.• libpmemlog, libpmemblk, libpmemobj libraries are built on top of libpmem and support transactions.