Anusha Ragunathan

Provisioning and Management

of Storage in the Docker

platform

Sr Software Engineer, Docker

Agenda

Docker Volume Plugins (DVP) on Engine &

Swarm

Docker Enterprise & Kubernetes

Storage Ecosystem

Storage Ops is Hard A key demand enterprises have is that developers should be able to store data

in clusters without having to worry about how persistent storage is working

under the hood.

Provision

Volumes

Deploy

Apps HA DR

Understand

Your

use-case

Map

use-case to

plugins

Life of a volume: Single node

CreateVolume

UnMountVolume (from

container)

MountVolume (to container

and format if necessary)

DeleteVolume (after use)

A way to extend the docker engine's volume subsystem on a

single node. An out-of-tree implementation for:

● Create/Remove volumes

● Mount/Unmount volumes

● Get Volume Capabilities

● List volumes

Examples: DVP include NetApp, VMware vSphere,

PureStorage plugins.

Docker Volume Plugins: Engine

Life of a volume: Cluster aware CreateVolume

AttachVolume (to Node)

Format and partition volume, if necessary

MountVolume (to container)

UnMountVolume (from container)

DetachVolume (from Node)

DeleteVolume (after use)

Docker Swarm does not orchestrate volume creation across a

cluster.

• Docker Swarm makes use of volume names instead of

volume ID as unique volume identifier

• Volume requests are simultaneously send to each node in a

Swarm cluster.

• Volume Plugins must run independently on each node in a

Swarm cluster.

These results in issues like race conditions that can result in

the creation of a large number of volumes for a single request

without a clear winner.

Docker Volume Plugins: Swarm

● Around the same time that Docker Swarm was considered

getting re-architected to be volume aware, a new industry

standard was evolving in the container orchestration

community.

● Container Storage Interface (CSI) started and matured by

contributions from Docker, Kubernetes and Mesos.

● Today, CSI specification has reached 1.0. Container

Orchestrators such as Kubernetes and Mesos have

supported for CSI.

● Docker Enterprise is a strong integrator for Kubernetes and

works well with CSI and storage providers.

Container Storage Interface

Docker Enterprise + Kubernetes

● A storage claim made by a

user

● Just like how Pods consume

Compute resources, PVC

consume Volume resources.

● Just like how Pods can

request specific levels CPU &

memory, PVCs can request

specific sizes and access

modes.

● Pods reference PVC

Persistent Volume

Claim (PVC)

Persistent Volume (PV)

● Storage resources in a cluster

● Lifecycle independent of a Pod

Pods

● A set of running containers

representing a workload

● Provisioning is the creation/allocation of Persistent Volumes

● Static and Dynamic Provisioning

○ Static: pre creation of PV

○ Dynamic: automatic creation of PV based on size,

permissions requested.

● Dynamic Provisioning through Storage Classes

○ Provides a way for Admins to describe “classes” of

storage available. Eg, different performance SLAs,

value-add features such as replication, backup, etc

○ Backed by a provisioner

PV Provisioning

● Provisioning is the creation/allocation of Persistent Volumes

● Static and Dynamic Provisioning

○ Static: pre creation of PV

○ Dynamic: automatic creation of PV based on size,

permissions requested.

● Dynamic Provisioning through Storage Classes

○ Provides a way for Admins to describe “classes” of

storage available. Eg, different performance SLAs,

value-add features such as replication, backup, etc

○ Backed by a provisioner

PV Provisioning

Provisioning Workflow (static)

1. Cluster Admin

pre-provisions

volumes and

registers PVs

2. Developer

claims a PV

from the pool

4. Developer

references the

claim in a Pod

3. Controller BINDS

PV to PVC

Pool of Persistent Volumes

NFS PV iSCSI PV EBS PV

Claim

`Claim

ClaimClaim

Pod

Claim

5. Controller inspects

Claim and MOUNTs

the PV into the Pod.

Provisioning Workflow (dynamic)

1. Cluster Admin

registers

Storage Classes

2. Developer defines a

claim by referring a

storage class

5. Developer

references the

claim in a Pod

4. Controller BINDS

PV to PVC

6. Controller

inspects Claim

and MOUNTs the

PV into the Pod.

SSD

Pod

Claim

Claim

Persistent

Volumes Storage

Classes

Slow

Fast

3. Controller

provisions

volumes

on-demand

Demo 1: PV Provisioning using iSCSI

Worker Worker

Master

ISCSI Target Server

High Availability (HA) ● HA involves handling Node failures and Storage failures.

● Node failures:

○ Node drain: Maintenance

○ Node failures: Node lost/ kubelet crash

● Kubernetes provides built-in support for node failures. The

attach-detach controller on the master is in charge of

interacting with the volume plugin and moving PVs across

nodes.

● If delays/instability with attach/detach is not desired, use a

Software Defined Storage (SDS) solution.

Plugin Type Pros Cons Future

InTree No extra installation

necessary

Release cycles tied to

k8s releases Superseded by CSI

External

Provisioner

Code maintained

independently

Limited customization for

attach and mount ops Superseded by CSI

FlexVolume Highly customizable code

maintained independently

Host based exec model

reduces portability

Deprecated for Linux

Will be used for Windows

CSI

Highly customizable code

maintained independently

and based on standard

Requires installation and

configuration

The future of storage

plugins

Kubernetes Storage Plugin Ecosystem

In-tree Plugin Scenarios

Cloud Protocol Ephemeral SDS Extensions

AWS EBS NFS local Portworx FlexVolume

GCE PD iSCSI hostpath StorageOS CSI

Azure Disk/File Fibre Channel config_map ScaleIO

Openstack Cinder secret

vSphereVolume

In tree Plugin Architecture

Worker Worker

Master

Storage Service

Kubelet Kubelet

AttachVolume

DetachVolume

CreateVolume

DeleteVolume

Kube Core Components

MountVolume

UnmountVolume

AttachVolume

DetachVolume

CreateVolume

Delete Volume

In-tree Plugins

PV Claim

API Server

External Provisioner Scenarios

In-tree PV source Implementations [not certified with UCP yet]

iSCSI Netapp-Trident, Dell/EMC-Isilon, HPE-Nimble, HPE-3PAR, Nutanix,

OpenEBS-iscsi

NFS Netapp-Trident, AWS-EFS, Dell-EMC Isilon

External Provisioner Architecture

Worker Worker

Master

Storage Service

Kubelet Kubelet

External Provisioner (StatefulSet/Deployment Pod)

Kube Core Components External components

API Server

MountVolume UnmountVolume

AttachVolume DetachVolume

CreateVolume

Delete Volume

In-tree Plugins

PV Claim

PV

CSI Scenarios

Cloud Local/ephemeral

On-prem SAN/NAS SDS

CSI Plugins

Worker Worker

Storage Service

Kubelet Kubelet

External

Provisioner (Deployment/

StatefulSet)

CreateVolume

DeleteVolume

External

Attacher (Deployment/

StatefulSet)

CSI Controller (Deployment/

StatefulSet)

CSI Node (DaemonSet)

Kube Core Components Kube Sidecar containers CSI plugin components

ControllerPublish

ControllerUnPublish

Master

API Server

NodeStage/NodeUnstage NodePublish/NodeUnpublish

PV

PV Claim

volume attachment

CSI In-

Tree

Future enterprise features in CSI

❖Backup

➢Snapshot/restore

➢Application consistent/triggered snapshots

❖Volume resizing

❖Cloning and Replication

Demo 2: PV Provisioning using CSI AWS EBS

Worker Worker

Master