1. Ceph 2014/6/5 Open Programmable Infrastructure Environment 14/06/02 Copyright 2014(C) OSS Laboratories Inc. All Rights Reserved 1

2. l l Disk Virtualization :(LBA) l Block Virtualization :LVM/RAID l File Virtualization VFS l File System Virtualization (SNIAStorage Network Industry Association) l l /Software Defined Storage, etc. l l l EMC ViPR l VMware VSAN l Datacore SANSymphony-V l NexentaStor l Cleversafe/Amplidata l 214/06/02Copyright 2014(C) OSS Laboratories Inc. All Rights Reserved 3. l l l 3 (REST)S3 EBS (REST, JSON/XML etc.)Cloud Storage Persistent Disk (REST, NTFS)BLob Storage(Azure Drive) Local Storage 14/06/02Copyright 2014(C) OSS Laboratories Inc. All Rights Reserved 4. 4 VMware, OpenStack, etc. POSIX (FUSE) VM REST API (S3, SWIFT) 14/06/02Copyright 2014(C) OSS Laboratories Inc. All Rights Reserved 5. 14/06/02Copyright 2014(C) OSS Laboratories Inc. All Rights Reserved 5 1 Amazon S3 Andrew File System (AFS) etc. Microsoft DFS Moose FS, etc. FraunhoferFS (FhGFS) PVFS/OrangeFS etc. Ceph GlusterFS XtreemFS etc. 6. 14/06/02Copyright 2014(C) OSS Laboratories Inc. All Rights Reserved 6 Amage Ceph Inktank LGPL2 Chiron FS luisfurquim@gmail.com GPL3 Cloudian CloudStore/Kosmosfs/Quantcastfs Quantcast Apache License 2.0 Cosmos Microsoft internal dCache DESY and others FraunhoferFS (FhGFS) Competence Center for High Performance Computing FhGFS license FS-Manager CDNetworks General Parallel File System(GPFS) IBM Gfarm file system / BSD GlusterFS Gluster, a company acquired by Red Hat GPL3 Google File System(GFS) Google Hadoop Distributed File System ASF, Cloudera, Pivot, Hortonworks, WANdisco, Intel Apache License 2.0 IBRIX Fusion IBRIX LeoFS Apache License 2.0 Lustre originally developed byCluster File System and currently supported by Intel(formerly Whamcloud) GPL MogileFS Danga Interactive GPL 7. l l l l ID l l POSIX/FUSE/Block/REST l l WORM/Transactional locking/Leasing l l Read only/Read/WriteCAPPAXOS l l l Self Healing 714/06/02Copyright 2014(C) OSS Laboratories Inc. All Rights Reserved 8. Ceph l l Ceph (UCSC) Sage Weil DreamHost / Inktank 2004 2010 Linux (2.6.34 ) l CephLGPL V2.1Ceph Enterprise Inktank l l InktankOpen Core GUISNMP Hyper-V l 20127 201456V0.80.1(Firefly) Linux3.4.203.6.6 WindowsMacOpenStack CloudStackIaaS 814/06/02Copyright 2014(C) OSS Laboratories Inc. All Rights Reserved 9. 2014/4/30 Red Hat to Acquire Inktank, Provider of Ceph 14/06/02Copyright 2014(C) OSS Laboratories Inc. All Rights Reserved 9 10. Ceph l l PCOS(Linux) l l l l l l POSIX l l REST(AWS S3/SWIFT) l WANRGW l RW(LAN 1014/06/02Copyright 2014(C) OSS Laboratories Inc. All Rights Reserved 11. RADOS Reliable, Autonomous, Distributed Object Store MDS MonitorsOSDs Ceph 11 CephFS POSIX Kernel-client FUSE-client Librados C, C++, java, Python, Ruby, PHP RADOS API RADOSGW (RADOS Gateway) AWS S3 OpenStack SWIFT REST API http RBD (RADOS Block Device) Linux RADOS API Kernel Client vfs MDS MonitorsOSDs MDS MonitorsOSDs API QEMU /KVM Kernel Client FUSE Client Ext4/xfs/btrfs vfs Ext4/xfs/btrfs 12. CephFS (Posix file system) kernel Device driver vfs FUSE File system User space Kernel space Ceph-fuse Client Application MDS Monitors OSDs vfs vfs User space Kernel space Ceph-kernel Client Application RADOS CephFS(V.0.80) 14/06/02Copyright 2014(C) OSS Laboratories Inc. All Rights Reserved 12 CephFS Kernel client CephFS FUSE client 13. QEMU/KVM Ceph RBD 14/06/02Copyright 2014(C) OSS Laboratories Inc. All Rights Reserved 13 RBD (RADOS Block Device) Kernel Client librbd VM Linux Kernel Client Hypervisor VM Linux Linux RBD Cache Linux Hypervisor VM OS Hypervisor QEMU/KVM 14. Ceph RADOSGW (Object Gateway) librgw File system Monitors OSDs vfs S3 API librados Application RADOS RADOSGW httpd SWIFT API 14/06/02Copyright 2014(C) OSS Laboratories Inc. All Rights Reserved 14 15. OpenStack 14/06/02Copyright 2014(C) OSS Laboratories Inc. All Rights Reserved 15 Volumes Ephemeral Copy-on-Write Snapshots 16. Keystone Keystone Server RADOSGW RESTful Object Store Query token Access with token Grant/revoke 17. Glance RBD Glance Server /etc/glance/glance-api.conf default_store=rbd rbd_store_user=glance rbd_store_pool=images Store, Download Image 18. Cinder/Nova RBD Cinder Server Libvirt (QEMU/KVM) VM Boot from volume VolumeImage Copy-on-write clone librbd Nova/compute 19. OpenStack+Ceph l OpenStack l MySQL l OS Ceph l Compute Ceph l Network l 14/06/02Copyright 2014(C) OSS Laboratories Inc. All Rights Reserved 19 20. Ceph 14/06/02Copyright 2014(C) OSS Laboratories Inc. All Rights Reserved 20 Virtualization and Private Cloud (traditional SAN/NAS) High Performance (traditional SAN) PerformanceCapacity NAS & Object Content Store (traditional NAS) Cloud Applications Traditional IT XaaS Compute Cloud Open Source Block XaaS Content Store Open Source NAS/Object Ceph Target Ceph Target 21. Multi-site Replication (RADOSGW) l l Read Only l l l AP(Eventually consistency) 2114/06/02Copyright 2014(C) OSS Laboratories Inc. All Rights Reserved 22. Erasure coding l Erasure Coding l Azure l 20% 40% () l Erasure Coding (2 ) 140%200% CPUHighLow 23. CEPH STORAGE CLUSTER Write ReadRead/Write SSD HDD 24. Ceph (as of 4/30) 14/06/02Copyright 2014(C) OSS Laboratories Inc. All Rights Reserved 24 1.2 2.0 CEPH CALAMARI PLUGINS May 2014 Q4 2014 2015 25. l l VM l CephFSRBD Object Storage l HDDNFS l HW 14/06/02Copyright 2014(C) OSS Laboratories Inc. All Rights Reserved 25 26. 1 DB 1 : 2 8 KB 64KB 2 1 : 0 64 KB 1MB 3 1 : 2 64 KB 512KB 4 BLOB 0 : 1 128 KB 10MB 14/06/02Copyright 2014(C) OSS Laboratories Inc. All Rights Reserved 26 NO. 1 10 10 2 100 10 3 100 100 4 1000 100 5 300 300 o fabric x5 10 OS Ceph 27. l HW l NW l SW l CentOS6.5kernel3.1 l Ceph 0.78 HWNW HDDHP ML110 G7 Intel Celeron G530 @ 2.40GHz 1p/2c HP 250GB SATA disk x1 1Gbps ext3 NFSNfs v4 Cephext4 RTT ms Mbps Nuttcp 0.418 928.5 14/06/02Copyright 2014(C) OSS Laboratories Inc. All Rights Reserved 27 28. (1) l KVMVM Ceph l 5 l CephFS kernel client l CephFS FUSE client l CephRBD l CephRBDKVM VM) l CephRBDLibrbd+QEMU/KVM) l CephCentOS6 kernel3.1 l VMKVMqcow2 l Ceph l pool 2 rbd(Block Device) l ceph osd pool set data size 3 l ceph osd pool set data min_size 2 l MonitorMDSOSD l OSD l OS/Ceph Mon Ceph pool MDS OSD Mon MDS OSD Mon MDS OSD fio fabric Ceph client Kvm host 14/06/02Copyright 2014(C) OSS Laboratories Inc. All Rights Reserved 28 29. (2) l NFS / fio fabric 14/06/02Copyright 2014(C) OSS Laboratories Inc. All Rights Reserved 29 NFS V4 fio fabric NFS client Kvm host SATA / NFS 30. VM OS Page cache o NFS NFS CephFS kernel client Page cache o QEMU CephFS FUSE client Page cache o QEMU CephRBD_VMKVM VM) RBD cache o QEMU CephRBD_BM RBD cache o CephRBD_QEMULibrbd+QEMU/KVM) Page cache o RBD cache o 14/06/02Copyright 2014(C) OSS Laboratories Inc. All Rights Reserved 30 31. 1 DB W1 : R2 8KB Block, 64KB File 14/06/02Copyright 2014(C) OSS Laboratories Inc. All Rights Reserved 31 DB W1 : R2 8 KB1 Block, 64KB File Read KB/s 1010 10010 100100 1000100 300300 Write KB/s 1010 10010 100100 1000100 300300 DB W1 : R2 8 KB1 Block, 64KB File NFS 1,478 2,162 2,523 2,515 1,220 2,149 1,347 2,315 8,610 2,043 NFS 609 890 1,021 1,023 574 997 635 1,065 4,015 969 DB W1 : R2 8 KB1 Block, 64KB File CephFS_Kernel CepFS_FUSE CephRBD_VM 292 4,223 2,052 252 4,358 1,955 194 2,950 1,172 185 2,690 1,145 197 2,137 1,357 CephFS_Kernel CepFS_FUSE CephRBD_VM 118 1,704 844 102 1,759 807 89 1,357 552 85 1,243 538 92 1,002 646 CephRBD_BM CephRBD_QEMU 1,763 1,932 2,752 1,980 1,277 1,441 1,391 2,325 1,118 2,938 CephRBD_BM CephRBD_QEMU 729 791 1,111 809 601 679 655 1,082 533 1,372 !" #$!!!" %$!!!" &$!!!" '$!!!" ($!!!" )$!!!" *$!!!" +$!!!" ,$!!!" #!$!!!" #!#! #!!#! #!!#!! #!!!#!! &!!&!! -./" 0123./4516718" 012./4.9/:" 0123;?" 0123;