Docker Networking in Swarm, Mesos and Kubernetes [Docker Meetup Santa Clara | April 2016]
Provisioning and Management of Storage in the Docker...
Transcript of Provisioning and Management of Storage in the Docker...
Anusha Ragunathan
Provisioning and Management
of Storage in the Docker
platform
Sr Software Engineer, Docker
Agenda
Docker Volume Plugins (DVP) on Engine &
Swarm
Docker Enterprise & Kubernetes
Storage Ecosystem
Storage Ops is HardA key demand enterprises have is that developers should be able to store data
in clusters without having to worry about how persistent storage is working
under the hood.
Provision
Volumes
Deploy
AppsHA DR
Understand
Your
use-case
Map
use-case to
plugins
Life of a volume: Single node
CreateVolume
UnMountVolume (from
container)
MountVolume (to container
and format if necessary)
DeleteVolume (after use)
A way to extend the docker engine's volume subsystem on a
single node. An out-of-tree implementation for:
● Create/Remove volumes
● Mount/Unmount volumes
● Get Volume Capabilities
● List volumes
Examples: DVP include NetApp, VMware vSphere,
PureStorage plugins.
Docker Volume Plugins: Engine
Life of a volume: Cluster awareCreateVolume
AttachVolume (to Node)
Format and partition volume, if necessary
MountVolume (to container)
UnMountVolume (from container)
DetachVolume (from Node)
DeleteVolume (after use)
Docker Swarm does not orchestrate volume creation across a
cluster.
• Docker Swarm makes use of volume names instead of
volume ID as unique volume identifier
• Volume requests are simultaneously send to each node in a
Swarm cluster.
• Volume Plugins must run independently on each node in a
Swarm cluster.
These results in issues like race conditions that can result in
the creation of a large number of volumes for a single request
without a clear winner.
Docker Volume Plugins: Swarm
● Around the same time that Docker Swarm was considered
getting re-architected to be volume aware, a new industry
standard was evolving in the container orchestration
community.
● Container Storage Interface (CSI) started and matured by
contributions from Docker, Kubernetes and Mesos.
● Today, CSI specification has reached 1.0. Container
Orchestrators such as Kubernetes and Mesos have
supported for CSI.
● Docker Enterprise is a strong integrator for Kubernetes and
works well with CSI and storage providers.
Container Storage Interface
Docker Enterprise + Kubernetes
● A storage claim made by a
user
● Just like how Pods consume
Compute resources, PVC
consume Volume resources.
● Just like how Pods can
request specific levels CPU &
memory, PVCs can request
specific sizes and access
modes.
● Pods reference PVC
Persistent Volume
Claim (PVC)
Persistent Volume (PV)
● Storage resources in a cluster
● Lifecycle independent of a Pod
Pods
● A set of running containers
representing a workload
● Provisioning is the creation/allocation of Persistent Volumes
● Static and Dynamic Provisioning
○ Static: pre creation of PV
○ Dynamic: automatic creation of PV based on size,
permissions requested.
● Dynamic Provisioning through Storage Classes
○ Provides a way for Admins to describe “classes” of
storage available. Eg, different performance SLAs,
value-add features such as replication, backup, etc
○ Backed by a provisioner
PV Provisioning
● Provisioning is the creation/allocation of Persistent Volumes
● Static and Dynamic Provisioning
○ Static: pre creation of PV
○ Dynamic: automatic creation of PV based on size,
permissions requested.
● Dynamic Provisioning through Storage Classes
○ Provides a way for Admins to describe “classes” of
storage available. Eg, different performance SLAs,
value-add features such as replication, backup, etc
○ Backed by a provisioner
PV Provisioning
Provisioning Workflow (static)
1. Cluster Admin
pre-provisions
volumes and
registers PVs
2. Developer
claims a PV
from the pool
4. Developer
references the
claim in a Pod
3. Controller BINDS
PV to PVC
Pool of Persistent Volumes
NFS PV iSCSI PV EBS PV
Claim
`Claim
ClaimClaim
Pod
Claim
5. Controller inspects
Claim and MOUNTs
the PV into the Pod.
Provisioning Workflow (dynamic)
1. Cluster Admin
registers
Storage Classes
2. Developer defines a
claim by referring a
storage class
5. Developer
references the
claim in a Pod
4. Controller BINDS
PV to PVC
6. Controller
inspects Claim
and MOUNTs the
PV into the Pod.
SSD
Pod
Claim
Claim
Persistent
VolumesStorage
Classes
Slow
Fast
3. Controller
provisions
volumes
on-demand
Demo 1: PV Provisioning using iSCSI
Worker Worker
Master
ISCSI Target Server
High Availability (HA)● HA involves handling Node failures and Storage failures.
● Node failures:
○ Node drain: Maintenance
○ Node failures: Node lost/ kubelet crash
● Kubernetes provides built-in support for node failures. The
attach-detach controller on the master is in charge of
interacting with the volume plugin and moving PVs across
nodes.
● If delays/instability with attach/detach is not desired, use a
Software Defined Storage (SDS) solution.
Plugin Type Pros Cons Future
InTreeNo extra installation
necessary
Release cycles tied to
k8s releasesSuperseded by CSI
External
Provisioner
Code maintained
independently
Limited customization for
attach and mount opsSuperseded by CSI
FlexVolumeHighly customizable code
maintained independently
Host based exec model
reduces portability
Deprecated for Linux
Will be used for Windows
CSI
Highly customizable code
maintained independently
and based on standard
Requires installation and
configuration
The future of storage
plugins
Kubernetes Storage Plugin Ecosystem
In-tree Plugin Scenarios
Cloud Protocol Ephemeral SDS Extensions
AWS EBS NFS local Portworx FlexVolume
GCE PD iSCSI hostpath StorageOS CSI
Azure Disk/File Fibre Channel config_map ScaleIO
Openstack Cinder secret
vSphereVolume
In tree Plugin Architecture
Worker Worker
Master
Storage Service
Kubelet Kubelet
AttachVolume
DetachVolume
CreateVolume
DeleteVolume
Kube Core Components
MountVolume
UnmountVolume
AttachVolume
DetachVolume
CreateVolume
Delete Volume
In-tree Plugins
PV Claim
API Server
External Provisioner Scenarios
In-tree PV source Implementations [not certified with UCP yet]
iSCSINetapp-Trident, Dell/EMC-Isilon, HPE-Nimble, HPE-3PAR, Nutanix,
OpenEBS-iscsi
NFS Netapp-Trident, AWS-EFS, Dell-EMC Isilon
External Provisioner Architecture
Worker Worker
Master
Storage Service
Kubelet Kubelet
External Provisioner (StatefulSet/Deployment Pod)
Kube Core ComponentsExternal components
API Server
MountVolumeUnmountVolume
AttachVolumeDetachVolume
CreateVolume
Delete Volume
In-tree Plugins
PV Claim
PV
CSI Scenarios
Cloud Local/ephemeral
On-prem SAN/NAS SDS
CSI Plugins
Worker Worker
Storage Service
Kubelet Kubelet
External
Provisioner(Deployment/
StatefulSet)
CreateVolume
DeleteVolume
External
Attacher (Deployment/
StatefulSet)
CSI Controller (Deployment/
StatefulSet)
CSI Node(DaemonSet)
Kube Core Components Kube Sidecar containers CSI plugin components
ControllerPublish
ControllerUnPublish
Master
API Server
NodeStage/NodeUnstage NodePublish/NodeUnpublish
PV
PV Claim
volumeattachment
CSI In-
Tree
Future enterprise features in CSI
❖Backup
➢Snapshot/restore
➢Application consistent/triggered snapshots
❖Volume resizing
❖Cloning and Replication
Demo 2: PV Provisioning using CSI AWS EBS
Worker Worker
Master
AWS EBS Service
OS specific considerations
● Cluster may support Linux and Windows nodes
● Cluster-scoped operations can be OS agnostic
▪ Create, Delete
▪ Attach, Detach
● Node-scoped operations need to be OS aware
▪ Device enumeration
▪ Format, Mount, Dismount
OS specific considerations
● File system support varies across OS
▪ NTFS and SMB in Windows
▪ ext, xfs and NFS in Linux
● Support for privileged containers absent in Windows
▪ Plugins running directly on host have no problems
▪ Fully containerized plugins cannot function
▪ Host proxy to support CSI plugins being investigated
Persistent storage needs for containerized workloads is available for different
environments supporting a variety of use cases
The plugin model is standardized through a common industry standard spec -
CSI
Certify as many plugins as possible with UCP
to provide more choice to our customers
Summary
Thank You!
● Creating and Deleting Volumes
○ Handled by external-provisioner sidecar
○ Watches PersistentVolumeClaim objects and triggers CreateVolume and
DeleteVolume operations.
● Attaching and Detaching
○ Handled by external-attacher sidecar, unlike the attach-detach controller on the kube
master in external provisioners.
○ Watches VolumeAttachment objects and triggers ControllerPublish and
ControllerUnpublish against a CSI endpoint
● Mounting and Unmounting (into a Node)
○ Triggered by external attacher
○ NodeStageVolume, NodeUnstageVolume
● Mounting and Unmounting (into a Pod)
○ Triggered by external attacher
○ NodePublishVolume, NodeUnpublishVolume
CSI Operations [backup details]
CSI: Create PVC request
Kube core
external provisioner
CSI controller
CreatePersistentVolumeClaim
API request
Reference CSI plugin as provisioner in claim
Watches ‘PVC’ objects; triggers ‘CreateVolume’ CSI API call
Dispatch request to appropriate provisioner
AWS EBS
Service
API call to actual storage service to Create
the Volumes
EBS Volume
Created
CSI: Pod refers to PVC
Kube core
external attacher
CreatePod API request; Pod refers to PVC
Refer CSI plugin as provisioner in PVC
Watches ‘VolumeAttachment’ objects; triggers
‘ControllerPublishVolume’ CSI API call
Dispatch request to appropriate provisioner
API call to storage service to ‘Attach’ the
Created Volume to a specific node.CSI Controller
CSI Node
1. ControllerPublishVolume CSI API
Call
2. NodeStageVolume CSI API Call
3. NodePublishVolume CSI API Call
Mounts the attached Volume to a
‘Staging-Path’. This path is a global
directory on the node;
Bind mounts volumes from ‘Staging-Path’
to ‘Target-Path’ in a Pod. This enables
multiple Pods to refer to the same volume.
External Provisioner Operations
Operation Invocation
Provision/De-provisionPV claim => External Provisioner => PV object with In-Tree volume source
(iSCSI/NFS)
Attach/Detach AD controller => PV’s volume src plugin interface
Mount/Unmount Kubelet volume manager => PV’s volume src plugin interface
In-tree Plugin Operations
Operation Invocation
Provision/De-provision PVC => PV controller => plugin interface
Attach/Detach AD controller => plugin interface
Mount/Unmount Kubelet volume manager => plugin interface
CSI Operations
Operation Invocation
Provision/De-provision PV claim => CSI External Provisioner => CSI Plugin => PV with CSI volume source
Attach/Detach CSI in-tree => VolumeAttachment => CSI External Attacher => CSI Plugin
Mount/Unmount Kubelet CSI client => CSI Plugin
Backup [Alpha] Volume Snapshot object => CSI External Snapshotter => CSI Plugin
Restore [Alpha] PV claim => CSI External Provisioner => CSI Plugin