Bridging On-premise File System and Cloud Storage - SNIA · 2015 Storage Developer Conference. ©...
Transcript of Bridging On-premise File System and Cloud Storage - SNIA · 2015 Storage Developer Conference. ©...
2015 Storage Developer Conference. © EMC. All Rights Reserved.
Bridging On-premise File System and Cloud Storage
Pankaj Datta EMC – Isilon Storage Division
2015 Storage Developer Conference. © EMC. All Rights Reserved.
Acknowledgements
To the amazing cloud ops team…
2
2015 Storage Developer Conference. © EMC. All Rights Reserved.
Outline
Motivation OneFS Overview Architecture Features Integration Q&A
3
2015 Storage Developer Conference. © EMC. All Rights Reserved.
Motivation
Cloud is always available, anywhere Use it as a low cost, virtually unlimited, storage
tier for data not needed immediately Use higher performance, local OneFS NAS, tiers
otherwise
4
2015 Storage Developer Conference. © EMC. All Rights Reserved.
OneFS Overview
5
Isilon IQ Storage Layer
Intracluster Communication
Infiniband
Servers
Client/Application Layer Ethernet Layer
Servers
Servers
2015 Storage Developer Conference. © EMC. All Rights Reserved.
<App>
Architecture: Cloudpools
6
Policy
Accounts
Cloud Adapter
Storage
KM
Pools
Cloud-IO
Cloud-Jobs
RAN ECS
Azure S3
Archive Recall GC
Invalidate Writeback
NFS CIFS HDFS Swift
Redirector
Stats
Cloud Libs
2015 Storage Developer Conference. © EMC. All Rights Reserved.
Features: Policy
Associated with a pool of cloud accounts Various criteria to select which content gets
stored (archived) to the cloud: Path, Changed-time, Size, Type, etc
Various actions to select how content gets handled: Compression, Encryption, Caching, Retention
7
2015 Storage Developer Conference. © EMC. All Rights Reserved.
Features: Archive
File metadata stays on OneFS cluster File data chunked and stored on the cloud: 1 Cloud Metadata Object (CMO) N Cloud Data Object (CDO)
Multiple concurrent file archiving File becomes ‘Smartlink’ after successful archive A new Smartlink has a tiny OneFS footprint
8
2015 Storage Developer Conference. © EMC. All Rights Reserved.
Features: What is a Smartlink?
9
MDS/Cache
Cacheinfo Mapinfo
ACLs
Att
rs
CMO
CDO CDO
AD
Ss Special OneFS file Pointer like Data bearing Versioned
Shared metadata
Checkpt
2015 Storage Developer Conference. © EMC. All Rights Reserved.
Features: Caching
Smartlink access can trigger caching Cached data stored inside Smartlink file Caching metadata (Cacheinfo) in file ADS Cache maintains state per file region Uncached, Cached, Dirty, Syncing
Caching behavior controlled by policy Fully cached Smartlink != regular file
10
2015 Storage Developer Conference. © EMC. All Rights Reserved.
Features: [Total!] Recall
Uncached data in Smartlink fetched from cloud Smartlink becomes regular file after successful
recall Multiple concurrent file recall Unreferenced cloud objects deleted later
11
2015 Storage Developer Conference. © EMC. All Rights Reserved.
Features: Cloud Data Model
12
Smartlink V1 Smartlink V2
2015 Storage Developer Conference. © EMC. All Rights Reserved.
Features: Writeback
Periodic job execution Dirty cache regions written to cloud Perform Smartlink cacheinfo and mapinfo
update
13
2015 Storage Developer Conference. © EMC. All Rights Reserved.
Features: Invalidation
Periodic job execution Addresses the need to ‘shrink’ Smartlink OneFS
footprint Smartlinks with ‘Cached’ regions are cleared of
local content LRU semantics
14
2015 Storage Developer Conference. © EMC. All Rights Reserved.
Features: Security
Policy configurable AES 256 encryption Encrypt data before storing to cloud Encryption keys from OneFS Key Manager (KM) Block based encryption
15
2015 Storage Developer Conference. © EMC. All Rights Reserved.
Features: Garbage Collection (GC)
Any CMO not referenced on OneFS is an orphan
An orphan has a Date of Death (DOD) dictated by retention times
[Cloud] GC orphan when its DOD expires Asynchronous, schedule driven, collection
16
2015 Storage Developer Conference. © EMC. All Rights Reserved.
Features: Cloud Libs
Provide APIs to handle Smartlinks Low level: detect, read, write, truncate, delete,
etc High Level: app specific
Integrate with OneFS apps (snapshot, backup, replication, etc)
17
2015 Storage Developer Conference. © EMC. All Rights Reserved.
Features: Stats
Collect statistics on cloud access from the cluster
Inform user or throttle operations when thresholds are exceeded
Support per minute, hour, day, month granularity of presentation
18
2015 Storage Developer Conference. © EMC. All Rights Reserved.
Features: Metering
Intent to understand how much cloud storage is being used
Usage data fetched for cloud accounts used by a cluster
Reported to Isilon support central
19
2015 Storage Developer Conference. © EMC. All Rights Reserved.
FS Integration
A look at Smartlink and OneFS integration
20
2015 Storage Developer Conference. © EMC. All Rights Reserved.
FS Integration: Protocol support
Smartlink accessible over NFS, CIFS, HDFS, and Swift just like a regular file
Redirect handling of Smartlink to cloudpools Return accessed blocks from local storage on
cache-hit Fetch accessed blocks from cloud on cache-
miss
21
2015 Storage Developer Conference. © EMC. All Rights Reserved.
FS Integration: Snapshots
Integrated into OneFS snapshots Smartlink snap represents point-in-time data
from OneFS and Cloud storage Support cloud COW by using provider specific
operations Snap-diff support for Smartlink
22
2015 Storage Developer Conference. © EMC. All Rights Reserved.
FS Integration: Backup
Integrate into OneFS NDMP backup – no change to backup application
Smartlink shared metadata (accounts, pool, policy, keys) backed (and restored) as well
Configurable backup retention time Option to backup Smartlink as regular file
23
2015 Storage Developer Conference. © EMC. All Rights Reserved.
FS Integration: Replication
Integrated into OneFS SyncIQ – bidirectional content synchronization across clusters
Smartlink replicated to target cluster in a fraction of time on average
Smartlink replication typically takes up miniscule network bandwidth – cluster to cluster (or cloud)
Smartlink can be replicated as regular file and any changes to either synced across clusters
24
2015 Storage Developer Conference. © EMC. All Rights Reserved.
FS Integration: Changelists
OneFS changelists allow customer identification of changed files across two snapshots
Prevent false positives for Smartlinks Cache read Cache invalidation
25
2015 Storage Developer Conference. © EMC. All Rights Reserved.
Q & A
26