Introduction and Beyond

Karan SinghSr. Storage Architect

Red Hat , FinlandKaran@redhat.com

•Author for Learning Ceph and Ceph CookBook•www.ksingh.co.in

# whoami

What is Software Defined Storage (SDS) ?

•Fancy name for Storage Virtualization•All the intelligence is built into software•Commodity Hardware•! (Big Iron , Secret , Black Boxes) in your DC

What are different storage options ?

FILE STORAGEFile systems allow users to organize data stored in blocks using hierarchical folders and files.

OBJECT STORAGEObject stores distribute data algorithmically throughout a cluster of media, without a rigid structure.

BLOCK STORAGEPhysical storage media appears to computers as a series of sequential blocks of a uniform size.

Ceph Introduction & Architecture

Why Ceph is Cool ?

• Open Source Software Defined Storage• Unified Storage Platform ( Block , Object and File Storage )• Runs on Commodity Hardware• Self Managing, Self Healing• Massively Scalable • No Single Point of failure•Designed for cloud infrastructure and emerging workloads•Awesome release names

CPU Memory HDDs SSD Network

Linux OS

Ceph

Ceph Storage Cluster

...

...

...

SERVER 1 SERVER 2 SERVER 3 SERVER N

Block StorageObject Storage File Storage

Clients

CPU Memory HDD SSD Network

Linux OS

Ceph

CPU Memory HDD SSD Network

Linux OS

Ceph

CPU Memory HDD SSD Network

Linux OS

Ceph

Commodity Hardware

Linux OS

Ceph SDS

Distributed Storage

Unified Storage Solution

Ceph Storage Cluster

LIBRADOSA library allowing apps to directly access RADOS (C, C++, Java, Python, Ruby

RADOSA software-based reliable, autonomous, distributed object store comprised of

self-healing, self-managing, intelligent storage nodes and lightweight monitors

RBDA reliable, fully distributed block device

with cloud platform integration

RGWA web services gateway for object

storage, compatible with S3 and Swift

APP HOST/VM

CephFSA distributed file system with POSIX

semantics & scale-out metadata

Ceph Architecture

OSDs (Object Storage Daemon)• 10s to 10000s in a cluster• Typically one daemon per physical HDD• Serve stored data to clients• Intelligently peer for replication & recovery

Monitors• Maintain cluster membership and state• Provide consensus for distributed decision-making• Small, odd number• Do not store data

Ceph Components

RADOS CLUSTER

APPLICATION

RADOS Cluster a.k.a Ceph Cluster

APPLICATION

OBJECTS

CRUSH ALGORITHM

LIBRADOS

How data is stored in Ceph

Ceph Access Methods

LIBRADOSA library allowing apps to directly access RADOS (C, C++, Java, Python, Ruby)

RADOSA software-based reliable, autonomous, distributed object store comprised of

self-healing, self-managing, intelligent storage nodes and lightweight monitors

RBDA reliable, fully distributed block device

with cloud platform integration

RGWA web services gateway for object

storage, compatible with S3 and Swift

APP HOST/VM

CephFSA distributed file system with POSIX

semantics & scale-out metadata

Librados : Native Cluster Access

APP

LIBRADOSA library allowing apps to directly access RADOS (C, C++, Java, Python, Ruby)

RADOSA software-based reliable, autonomous, distributed object store comprised of

self-healing, self-managing, intelligent storage nodes and lightweight monitors

RBDA reliable, fully distributed block device

with cloud platform integration

RGWA web services gateway for object

storage, compatible with S3 and Swift

APP HOST/VM

CephFSA distributed file system with POSIX

semantics & scale-out metadata

Librados : Base Access Layer

LIBRADOSA library allowing apps to directly access RADOS (C, C++, Java, Python, Ruby)

RADOSA software-based reliable, autonomous, distributed object store comprised of

self-healing, self-managing, intelligent storage nodes and lightweight monitors

RBDA reliable, fully distributed block device

with cloud platform integration

RGWA web services gateway for object

storage, compatible with S3 and Swift

APP HOST/VM

CephFSA distributed file system with POSIX

semantics & scale-out metadata

RBD : Block Storage Access

VMHYPERVISOR

LIBRBD

HYPERVISOR

LIBRBD

RADOS CLUSTER

RBD : Separating Compute from Storage

LIBRADOSA library allowing apps to directly access RADOS (C, C++, Java, Python, Ruby)

RADOSA software-based reliable, autonomous, distributed object store comprised of

self-healing, self-managing, intelligent storage nodes and lightweight monitors

RBDA reliable, fully distributed block device

with cloud platform integration

RGWA web services gateway for object

storage, compatible with S3 and Swift

APP HOST/VM

CephFSA distributed file system with POSIX

semantics & scale-out metadata

Ceph Rados Gateway : Object Storage Access

RADOSGWRADOSGW

LIBRADOS

REST

Socket

CEPH CLUSTER

LIBRADOS

Ceph Rados Gateway : Object Storage Access

LIBRADOSA library allowing apps to directly access RADOS (C, C++, Java, Python, Ruby)

RADOSA software-based reliable, autonomous, distributed object store comprised of

self-healing, self-managing, intelligent storage nodes and lightweight monitors

RBDA reliable, fully distributed block device

with cloud platform integration

RGWA web services gateway for object

storage, compatible with S3 and Swift

APP HOST/VM

CephFSA distributed file system with POSIX

semantics & scale-out metadata

CephFS : File Storage

Linux Host

LIBCEPHFS

CEPH CLUSTER

CephFS : File Storage

Ceph Use Cases

GLANCE SWIFT NOVA CINDER MANILA

RADOSGW

INSTANCE IMAGE

LIBRBD

SWIFT OBJECT BOOT IMAGE

LIBRBD

BLOCK STORAGELIBRBD

FILE SHARE

CEPHFS

STOREGETPUT...

BOOT BLOCK DEVICE MOUNT

The complete storage solution for OpenStack

CEPH CLUSTER

Web Scale Object Storage

RGW RGW RGW RGW RGW

Load Balancer Load Balancer

Keystone

Neutron Nova

Cinder

Horizon

RabbitMQ

OpenStack Controller

Ceph Monitor

Glance

OSPd, Ceph installer

nova compute, tenant VMs, ceph-osd

MariaDB-Galera

NIC3

NIC2NIC1

NIC1

NIC1

NIC2NIC4

NIC3

NIC4

HCI : OpenStack and Ceph

https://goo.gl/8gv5un

Ceph : What's New ?

BlueStore : New OSD Backend

● BlueStore = Block + New Store○ KV DB (rocksdb) for metadata○ All data written directly to raw device○ Can combine HDD, SSD , NVMe

● Replaces FileStore, which was designed for HDDs● Eliminates the need for an underlying filesystem or dedicated journal device● Provides a 2-3X performance boost● Full data checksum ( CRC32, hash )● Inline Compression ( zlib , snappy , zstc)

Blue StoreCompressionOver The Wire encryptionManagement

Ceph-mgrTendrl GUI Management

ConsoleQoSCeph in ContainersRBD

Erasure Coding SupportVolume MirroringiSCSI

Ceph : What's New ?

RGWRGW Metadata IndexingLDAP / ADKeystone v3Object expirationStatic WebsitesNFS Gateway for RGW

MDSActive / Active MDS

support for CephFSRepair and DR toolsOpenStack Manila

Support