Leases and-caching final
-
Upload
glusterorg -
Category
Technology
-
view
105 -
download
0
Transcript of Leases and-caching final
Soumya KoduriPoornima G
Lease Locks and Caching
Gluster design SummitMay. 2015
Agenda
Quick intro to Lease locks
Why implement in Gluster
Design
Network Partitions
Challenges / limitations
Future enhancements
Client side caching with leases
Locking mechanism, which guarentees the client, certain semantics with respect to the sharing of the file/directory with other clients.
Granted on regular files (file/directory).
Protocol for cache consistency and aggressive caching.
Enhances performance.
Quick intro to Lease locks
Quick intro to Lease locks
Client 1Server
Request for lease on file1file1file1
Granted,cache file1Client 2
Request for access to file1 Lease Break file1Flush all the cached data
Flush file1,Unlock lease
6.Granted accessto file1126435
Why support it in Gluster
NFS-Ganesha/Samba applications
Multi-protocol access on Gluster
A new client side caching xlator based on leasesFile data
Directory entries (+ readdirplus)
Metadata
Lease locks and its manifestations
Linux lease locks:- Read lease- Write lease- Process based
NFS V4 delegations:- Read delegations- Write delegations- Directory delegations- Client based
Lease locks and its manifestations
SMB- Oplocks (opurtunistic locking)
Level 1 (exclusive lock) Level 2 (Read lock) Batch (handle caching + exclusive lock) Fd based
Leases introduced in SMB 2.1 Read Read/Handle Read/Write Read/Write/handles Client based
Oplocks was introduced in SMB 2?Leases were introduced in SMB 2.1 considered as oplocks done right. Major differences being:- Introduction of new typrs of leases- Introduction of lease Key
On Gluster
Currently supported- Read Lease- Write Lease- fd based
Plan is to support a super set of all standards:- Handle Lease, Read handle, Read Write Handle Lease- Lease Key support i.e. client based.- Lease upgrade/downgrade.- Directory Lease
Doesn't adhere to the Posix standard of one unlock per process.
Design
Gluster Server
Locks xlatorLease xlator
libgfapiProtocol clntProtocol server
glfs_adv_lock()
Lk Request
Lk Response
Recall LeaseUses Upcall Infra- Stores the lease info
- Check conflict,grant/ Recall lease
- Timer based recall
Command to enable Lease lock support for a volume:Gluster volume set leaeses on
Recall Lease implemented via the file system notification feature in Gluster
The lease lock requests are not sent to the Posix locks xlator, and hence it should be placed above the Posix locks, for it to function right.
Design
New API introduced in libgfapi to request lease locks.int glfs_*advanced*_lock ();
Lease xlator on server side above Posix locks:- Maintains the lease state of a file- Checks for conflicts, Grants, Recall, lease locks
Recall lease is implemented using Gluster file system notifications.
Timer based recall
Command to enable Lease lock support for a volume:Gluster volume set leaeses on
Recall Lease implemented via the file system notification feature in Gluster
The lease lock requests are not sent to the Posix locks xlator, and hence it should be placed above the Posix locks, for it to function right.
Network Parition
Gluster Server disconnects:
Replica brick will serve the lease state.
Non-replica - the lease state is lost forever.
Gluster Client disconnects:
All the locks held by this client will be unlocked.
Gluster Server/Client Reconnects:
The client will replay all the lease locks it has held.
Command to enable Lease lock support for a volume:Gluster volume set leaeses on
Recall Lease implemented via the file system notification feature in Gluster
The lease lock requests are not sent to the Posix locks xlator, and hence it should be placed above the Posix locks, for it to function right.
Challenges / Limitations
Rebalance/Tiering
Self-Heal
Client lock replay not enabled.
Inconsistency when used along with performance xlators
Delegation throttling
Tracking non fd based fops and the actual client associated
Future Enhancements
Super set of all leases
Upgrade and downgrade leases
Heuristics based lease grantConflict frequency
Limit on leases granted
Quota limits
Migrate/heal lease state
Locks recovery during network disconnects
Client side caching
Common challenges of caching
Challenges with caching
Mostly in memory hence less aggressive
Not distributed cache coherent
Performance hit with cache-validation requests
Not tolerant with file-system notification failures
architectures; this work relies on reliable, synchronousbroadcast communication as provided by the system bus.A distributed system, however, can experience partial fail-ures: a host may crash or messages may be lost. Existingapproaches to consistency for file caches fall into two cat-egories: those that assume reliable broadcast, and so donot tolerate communication failures, and those that requirea consistency check for every read, and so fail to delivergood performance.
Benifits of using leases
Cache Coherency
Aggressive persistent data caching
Reduces application-level response time
Suitable for distributed systems of larger scale with their larger ratio of processor speed to network delay
Performance improvement in the workloads with little write-sharing
architectures; this work relies on reliable, synchronousbroadcast communication as provided by the system bus.A distributed system, however, can experience partial fail-ures: a host may crash or messages may be lost. Existingapproaches to consistency for file caches fall into two cat-egories: those that assume reliable broadcast, and so donot tolerate communication failures, and those that requirea consistency check for every read, and so fail to delivergood performance.
Benifits of using leases
Directory leases will help in caching readdir and readdirplus coherently
Well suited for home directory use cases
Decreases file creation latency
Client cache based on leases
Gluster Server
Locks xlatorLease xlator
Libgfapi/FUSEProtocol clntProtocol server
Open file1
Open + Lease req
Lease Granted rsp
Lease Cache
Open
Open+lease
Cache file1 if lease grantedGluster Client
Command to enable Lease lock support for a volume:Gluster volume set leaeses on
Recall Lease implemented via the file system notification feature in Gluster
The lease lock requests are not sent to the Posix locks xlator, and hence it should be placed above the Posix locks, for it to function right.
Thank You!
GlusterFS Meetup
GlusterFS Meetup
Red Hat Proprietary
Click to edit the title text formatClick to edit Master title style
Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline LevelSeventh Outline LevelClick to edit Master text stylesSecond levelThird levelFourth level
Fifth level