Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely...
Transcript of Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely...
![Page 1: Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely multiplex SCM . User. Kernel. HW. API . APP . LibFS (layout, logic) API . APP . OtherLibFS](https://reader031.fdocuments.net/reader031/viewer/2022013003/5f075cef7e708231d41c9c66/html5/thumbnails/1.jpg)
Aerie: Flexible File-System Interfaces to Storage-Class Memory
Haris Volos† Sanketh Nalli, Sankaralingam Panneerselvam, Venkatanathan Varadarajan, Prashant Saxena,
Michael M. Swift
HP Labs †
![Page 2: Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely multiplex SCM . User. Kernel. HW. API . APP . LibFS (layout, logic) API . APP . OtherLibFS](https://reader031.fdocuments.net/reader031/viewer/2022013003/5f075cef7e708231d41c9c66/html5/thumbnails/2.jpg)
• Persistent • Short access time
2
Software overhead matters
Storage-Class Memory (SCM) Flash-backed DRAM
Flash
Phase-Change Memory
Spin Torque MRAM Resistive RAM
ns μs
Latency
![Page 3: Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely multiplex SCM . User. Kernel. HW. API . APP . LibFS (layout, logic) API . APP . OtherLibFS](https://reader031.fdocuments.net/reader031/viewer/2022013003/5f075cef7e708231d41c9c66/html5/thumbnails/3.jpg)
3
Memory Controller
SCM DRAM
Storage-Class Memory (SCM)
• Persistent • Short access time • Byte addressable
Accessible via loads/stores
No software overhead
![Page 4: Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely multiplex SCM . User. Kernel. HW. API . APP . LibFS (layout, logic) API . APP . OtherLibFS](https://reader031.fdocuments.net/reader031/viewer/2022013003/5f075cef7e708231d41c9c66/html5/thumbnails/4.jpg)
• Direct user-mode access for fast access to data – Moneta-D, PMFS, Quill,
NV-Heaps, Mnemosyne
• File system for sharing
– Shared namespace – Protection – Integrity
4
OS/ File System
Application
Accessing SCM today
+
![Page 5: Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely multiplex SCM . User. Kernel. HW. API . APP . LibFS (layout, logic) API . APP . OtherLibFS](https://reader031.fdocuments.net/reader031/viewer/2022013003/5f075cef7e708231d41c9c66/html5/thumbnails/5.jpg)
Device Driver Device Driver
Generic Block Layer
I/O Scheduler
SCM FS
App
Virtual File System
App
Disk FS
I/O Bus
Does SCM need a kernel FS?
5
MMU protects CPU access
DMA is not protected
Variable latency to disk
~ Constant latency to SCM
Load/store interface
No standard interface
SCM Disk
![Page 6: Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely multiplex SCM . User. Kernel. HW. API . APP . LibFS (layout, logic) API . APP . OtherLibFS](https://reader031.fdocuments.net/reader031/viewer/2022013003/5f075cef7e708231d41c9c66/html5/thumbnails/6.jpg)
• Enable implementation flexibility – Optimize file-system interface semantics – Optimize operations regarding metadata
Library file systems (libFS)
6
APP
LibFS API
[Exokernel (MIT), Nemesis (Cambridge)]
![Page 7: Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely multiplex SCM . User. Kernel. HW. API . APP . LibFS (layout, logic) API . APP . OtherLibFS](https://reader031.fdocuments.net/reader031/viewer/2022013003/5f075cef7e708231d41c9c66/html5/thumbnails/7.jpg)
Aerie libFS in a nutshell
7
APP
LibFS (functionality)
Safely multiplex SCM
User
Kernel HW
API
APP
LibFS (layout, logic)
API
APP
OtherLibFS (layout, logic)
API
![Page 8: Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely multiplex SCM . User. Kernel. HW. API . APP . LibFS (layout, logic) API . APP . OtherLibFS](https://reader031.fdocuments.net/reader031/viewer/2022013003/5f075cef7e708231d41c9c66/html5/thumbnails/8.jpg)
APP APP
LibFS (layout, logic)
API
APP
OtherLibFS (layout, logic)
API
Safely multiplex SCM
8
API
User
HW
/
common bob alice
common alice bob /
Aerie libFS in a nutshell
open
Kernel
![Page 9: Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely multiplex SCM . User. Kernel. HW. API . APP . LibFS (layout, logic) API . APP . OtherLibFS](https://reader031.fdocuments.net/reader031/viewer/2022013003/5f075cef7e708231d41c9c66/html5/thumbnails/9.jpg)
Outline
• Overview • Motivation: Interface flexibility • Aerie: In-memory library file systems • Evaluation • Conclusion
9
![Page 10: Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely multiplex SCM . User. Kernel. HW. API . APP . LibFS (layout, logic) API . APP . OtherLibFS](https://reader031.fdocuments.net/reader031/viewer/2022013003/5f075cef7e708231d41c9c66/html5/thumbnails/10.jpg)
• Universal abstraction: Everything is a file – Has generic-overhead cost
10
Application
POSIX File (Virtual File System)
Storage File IPC Network
Socket
POSIX File: Expensive abstraction
![Page 11: Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely multiplex SCM . User. Kernel. HW. API . APP . LibFS (layout, logic) API . APP . OtherLibFS](https://reader031.fdocuments.net/reader031/viewer/2022013003/5f075cef7e708231d41c9c66/html5/thumbnails/11.jpg)
• Rigid interface and policies – Has fixed components and costs – Hinders application-specific customization
11
Application
POSIX File (Virtual File System)
UNIX concurrency semantics
Hierarchical names
Byte streams
Permissions
POSIX File: Expensive abstraction
![Page 12: Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely multiplex SCM . User. Kernel. HW. API . APP . LibFS (layout, logic) API . APP . OtherLibFS](https://reader031.fdocuments.net/reader031/viewer/2022013003/5f075cef7e708231d41c9c66/html5/thumbnails/12.jpg)
POSIX File: Expensive abstraction
• Rigid interface and policies – Has fixed components and costs – Hinders application-specific customization
Application
POSIX File (Virtual File System)
open() Syscall Naming + Permissions
In-memory state (Byte streams)
Concurrency control
File Descriptors
~ 2.5 μs ≈ 25x SCM latency
![Page 13: Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely multiplex SCM . User. Kernel. HW. API . APP . LibFS (layout, logic) API . APP . OtherLibFS](https://reader031.fdocuments.net/reader031/viewer/2022013003/5f075cef7e708231d41c9c66/html5/thumbnails/13.jpg)
File System
Motivating Example: Web Proxy
13
Web Proxy Cache
/cache
Characteristics • Flat namespace • Immutable files • Infrequent sharing
![Page 14: Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely multiplex SCM . User. Kernel. HW. API . APP . LibFS (layout, logic) API . APP . OtherLibFS](https://reader031.fdocuments.net/reader031/viewer/2022013003/5f075cef7e708231d41c9c66/html5/thumbnails/14.jpg)
Motivating Example: Web Proxy
14
Web Proxy Cache
Proxy FS POSIX FS
![Page 15: Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely multiplex SCM . User. Kernel. HW. API . APP . LibFS (layout, logic) API . APP . OtherLibFS](https://reader031.fdocuments.net/reader031/viewer/2022013003/5f075cef7e708231d41c9c66/html5/thumbnails/15.jpg)
Customizing the file system today
• Modify the kernel
• Add a layer over existing kernel file system • Use a user-mode framework such as FUSE
15
Cumbersome options
![Page 16: Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely multiplex SCM . User. Kernel. HW. API . APP . LibFS (layout, logic) API . APP . OtherLibFS](https://reader031.fdocuments.net/reader031/viewer/2022013003/5f075cef7e708231d41c9c66/html5/thumbnails/16.jpg)
• Software interface overheads handicap fast SCM
• Flexible interface is a must for fast SCM
• Library file systems can help remove generic software overheads
16
Flexible interfaces more important than ever
![Page 17: Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely multiplex SCM . User. Kernel. HW. API . APP . LibFS (layout, logic) API . APP . OtherLibFS](https://reader031.fdocuments.net/reader031/viewer/2022013003/5f075cef7e708231d41c9c66/html5/thumbnails/17.jpg)
Outline
• Overview • Motivation: Interface flexibility • Aerie: In-memory library file systems (libFS) • Evaluation • Conclusion
17
![Page 18: Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely multiplex SCM . User. Kernel. HW. API . APP . LibFS (layout, logic) API . APP . OtherLibFS](https://reader031.fdocuments.net/reader031/viewer/2022013003/5f075cef7e708231d41c9c66/html5/thumbnails/18.jpg)
Kernel safely multiplexes SCM
18
allocation, protection, addressing
Kernel HW
• Allocation: Allocates SCM regions (i.e. extents)
• Protection: Keeps track of region access rights
• Addressing: Memory-maps SCM regions
<extent>
![Page 19: Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely multiplex SCM . User. Kernel. HW. API . APP . LibFS (layout, logic) API . APP . OtherLibFS](https://reader031.fdocuments.net/reader031/viewer/2022013003/5f075cef7e708231d41c9c66/html5/thumbnails/19.jpg)
Library implements functionality
19
APP
LibFS (layout, logic)
allocation, protection, addressing
User
Kernel HW
API
APP
LibFS (layout, logic)
API
APP
OtherLibFS (layout, logic)
API
![Page 20: Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely multiplex SCM . User. Kernel. HW. API . APP . LibFS (layout, logic) API . APP . OtherLibFS](https://reader031.fdocuments.net/reader031/viewer/2022013003/5f075cef7e708231d41c9c66/html5/thumbnails/20.jpg)
Implementing file-system features
• File-system objects
• Shared namespace
• Protection (access control)
• Integrity
20
![Page 21: Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely multiplex SCM . User. Kernel. HW. API . APP . LibFS (layout, logic) API . APP . OtherLibFS](https://reader031.fdocuments.net/reader031/viewer/2022013003/5f075cef7e708231d41c9c66/html5/thumbnails/21.jpg)
File-system objects build on SCM extents
• Collection (or directory) – key → object ID (oid)
• mFile (or memory file) – Offset → data extent ID
21
![Page 22: Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely multiplex SCM . User. Kernel. HW. API . APP . LibFS (layout, logic) API . APP . OtherLibFS](https://reader031.fdocuments.net/reader031/viewer/2022013003/5f075cef7e708231d41c9c66/html5/thumbnails/22.jpg)
Shared namespace
22
APP
LibFS (layout, logic)
allocation, protection, addressing
User
Kernel HW
API
APP
LibFS (layout, logic)
API
APP
OtherLibFS (layout, logic)
API
common alice bob /
![Page 23: Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely multiplex SCM . User. Kernel. HW. API . APP . LibFS (layout, logic) API . APP . OtherLibFS](https://reader031.fdocuments.net/reader031/viewer/2022013003/5f075cef7e708231d41c9c66/html5/thumbnails/23.jpg)
23
APP
libFS
User
HW
APP
/
common bob alice
/
common bob alice
common alice bob /
Shared namespace
libFS
open
![Page 24: Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely multiplex SCM . User. Kernel. HW. API . APP . LibFS (layout, logic) API . APP . OtherLibFS](https://reader031.fdocuments.net/reader031/viewer/2022013003/5f075cef7e708231d41c9c66/html5/thumbnails/24.jpg)
24
APP
libFS
User
HW
APP
common alice bob /
Decentralize access control via hardware-enforced permissions
libFS /
common bob alice
/
common bob alice Memory protection prevents Bob from accessing Alice’s files
![Page 25: Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely multiplex SCM . User. Kernel. HW. API . APP . LibFS (layout, logic) API . APP . OtherLibFS](https://reader031.fdocuments.net/reader031/viewer/2022013003/5f075cef7e708231d41c9c66/html5/thumbnails/25.jpg)
25
APP
libFS
User
HW
APP
libFS /
shared alice
alice bob /
Hardware protection cannot guarantee integrity
common
/
common bob
foo
bar
![Page 26: Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely multiplex SCM . User. Kernel. HW. API . APP . LibFS (layout, logic) API . APP . OtherLibFS](https://reader031.fdocuments.net/reader031/viewer/2022013003/5f075cef7e708231d41c9c66/html5/thumbnails/26.jpg)
Trusted FS
Service (TFS)
26
APP
LibFS (layout, logic)
API
User
Kernel HW
APP
LibFS (layout, logic)
API
Integrity via Trusted File Service
allocation, protection, addressing
![Page 27: Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely multiplex SCM . User. Kernel. HW. API . APP . LibFS (layout, logic) API . APP . OtherLibFS](https://reader031.fdocuments.net/reader031/viewer/2022013003/5f075cef7e708231d41c9c66/html5/thumbnails/27.jpg)
27
HW
SCM
APP
LibFS API
File data Metadata
Read/ Write Read
Read/Write
APP
LibFS (layout, logic)
API
User
Trusted FS
Service (TFS)
Metadata Server (integrity)
Update
Decentralized architecture
RPC
![Page 28: Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely multiplex SCM . User. Kernel. HW. API . APP . LibFS (layout, logic) API . APP . OtherLibFS](https://reader031.fdocuments.net/reader031/viewer/2022013003/5f075cef7e708231d41c9c66/html5/thumbnails/28.jpg)
28
User
HW
SCM
APP
LibFS API
File data Metadata
Read/ Write Read
Read/Write
APP
API
Trusted FS
Service (TFS)
Metadata Server (integrity)
Lease Manager
(sharing) Log Update
Reducing communication: Hierarchical leases + Batching
RPC
![Page 29: Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely multiplex SCM . User. Kernel. HW. API . APP . LibFS (layout, logic) API . APP . OtherLibFS](https://reader031.fdocuments.net/reader031/viewer/2022013003/5f075cef7e708231d41c9c66/html5/thumbnails/29.jpg)
29
User
HW
SCM
APP
LibFS API
File data Metadata
Read/ Write Read
Read/Write
APP
API
Trusted FS
Service (TFS)
Metadata Server (integrity)
Lease Manager
(sharing) Log Free blocks
Update
Reducing communication: Hierarchical leases + Batching
RPC
common
adjust
Contention
![Page 30: Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely multiplex SCM . User. Kernel. HW. API . APP . LibFS (layout, logic) API . APP . OtherLibFS](https://reader031.fdocuments.net/reader031/viewer/2022013003/5f075cef7e708231d41c9c66/html5/thumbnails/30.jpg)
30
User
HW
SCM
APP
LibFS API
File data Metadata
Read/ Write Read
Read/Write
APP
API
Trusted FS
Service (TFS)
Metadata Server (integrity)
Lease Manager
(sharing) Log Free blocks
Update
Reducing communication: Hierarchical leases + Batching
RPC
![Page 31: Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely multiplex SCM . User. Kernel. HW. API . APP . LibFS (layout, logic) API . APP . OtherLibFS](https://reader031.fdocuments.net/reader031/viewer/2022013003/5f075cef7e708231d41c9c66/html5/thumbnails/31.jpg)
Prototype Implementation
• Extent API by Linux 3.2.2 x86-64 kernel modifications
• Communication via loopback RPC • Crash consistency through
– x86 CLFLUSH instruction (cache line flush) – Redo logging
• SCM emulation using DRAM
31
![Page 32: Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely multiplex SCM . User. Kernel. HW. API . APP . LibFS (layout, logic) API . APP . OtherLibFS](https://reader031.fdocuments.net/reader031/viewer/2022013003/5f075cef7e708231d41c9c66/html5/thumbnails/32.jpg)
32
HW
SCM
APP
LibFS libFS
File data Metadata
WRITE READ (/common)
APP
Trusted FS
Service (TFS)
Metadata Server (integrity)
Lease Manager
(sharing) Log Free blocks
Example: A shared file
creat(/common/foo) LOCK (/common)
CreateFile
LinkBlock
write(...)
RPC
![Page 33: Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely multiplex SCM . User. Kernel. HW. API . APP . LibFS (layout, logic) API . APP . OtherLibFS](https://reader031.fdocuments.net/reader031/viewer/2022013003/5f075cef7e708231d41c9c66/html5/thumbnails/33.jpg)
33
HW
SCM File data Metadata
RELEASE (/shared)
Trusted FS
Service (TFS)
Metadata Server (integrity)
Lease Manager
(sharing)
Example: A shared file
LOCK (/common)
APP
LibFS libFS
APP
LibFS
read(/common/foo)
READ (/common/foo) READ (/common)
API RPC
LOCK (foo)
![Page 34: Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely multiplex SCM . User. Kernel. HW. API . APP . LibFS (layout, logic) API . APP . OtherLibFS](https://reader031.fdocuments.net/reader031/viewer/2022013003/5f075cef7e708231d41c9c66/html5/thumbnails/34.jpg)
File Systems
Functionality: PXFS • POSIX interface:
open/read/write/unlink • Hierarchical namespace • POSIX concurrency
semantics • File byte streams
34
![Page 35: Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely multiplex SCM . User. Kernel. HW. API . APP . LibFS (layout, logic) API . APP . OtherLibFS](https://reader031.fdocuments.net/reader031/viewer/2022013003/5f075cef7e708231d41c9c66/html5/thumbnails/35.jpg)
File Systems
Functionality: PXFS • POSIX interface:
open/read/write/unlink • Hierarchical namespace • POSIX concurrency
semantics • File byte streams
Optimization: FlatFS • Key-value interface:
put/get/erase • Flat namespace
– Simplifies name resolution
• KV-store concurrency semantics – Reduce in-memory state
• Short, immutable files – Simplify storage allocation
35
![Page 36: Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely multiplex SCM . User. Kernel. HW. API . APP . LibFS (layout, logic) API . APP . OtherLibFS](https://reader031.fdocuments.net/reader031/viewer/2022013003/5f075cef7e708231d41c9c66/html5/thumbnails/36.jpg)
File Systems
36
PXFS
APP
FlatFS
APP
![Page 37: Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely multiplex SCM . User. Kernel. HW. API . APP . LibFS (layout, logic) API . APP . OtherLibFS](https://reader031.fdocuments.net/reader031/viewer/2022013003/5f075cef7e708231d41c9c66/html5/thumbnails/37.jpg)
Performance Evaluation
• Performance model – Writes to DRAM + software created delay – Reads to DRAM
• Configurations – RamFS: In-memory kernel FS – Ext4: ext4fs + RAM-disk – LibFS: PXFS and FlatFS
• Filebench workloads: Fileserver, Webserver, Webproxy
37
![Page 38: Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely multiplex SCM . User. Kernel. HW. API . APP . LibFS (layout, logic) API . APP . OtherLibFS](https://reader031.fdocuments.net/reader031/viewer/2022013003/5f075cef7e708231d41c9c66/html5/thumbnails/38.jpg)
Application-workload performance
0
5
10
15
20
Fileserver Webserver Webproxy
Late
ncy
per o
p
( μs
)
RamFS
ext4
PXFS
FlatFS
38
• PXFS performs better than kernel-mode FS • FlatFS exploits app semantics to improve performance
10%
9% 22%
53% (2.1x) 30%
![Page 39: Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely multiplex SCM . User. Kernel. HW. API . APP . LibFS (layout, logic) API . APP . OtherLibFS](https://reader031.fdocuments.net/reader031/viewer/2022013003/5f075cef7e708231d41c9c66/html5/thumbnails/39.jpg)
1
10
Late
ncy
per o
p ( μs
)
PXFS
ext4
FlatFS
Sensitivity to SCM performance: Webproxy
• Shorter SCM latencies favor – Direct access via load/store instructions – Interface specialization
39
0 100 1000 10000 Extra software delay ( ns )
![Page 40: Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely multiplex SCM . User. Kernel. HW. API . APP . LibFS (layout, logic) API . APP . OtherLibFS](https://reader031.fdocuments.net/reader031/viewer/2022013003/5f075cef7e708231d41c9c66/html5/thumbnails/40.jpg)
Scalability: Webproxy
0200400600800
10001200
0 5 10
Thro
ughp
ut (K
ops/
s)
# Threads
PXFSRamFSext4FlatFS
• FlatFS retains its benefits over kernel-mode file systems
Machine: Intel 6-core 2-way HT
![Page 41: Aerie: Flexible File -System Interfaces to Storage-Class ...€¦ · (functionality) Safely multiplex SCM . User. Kernel. HW. API . APP . LibFS (layout, logic) API . APP . OtherLibFS](https://reader031.fdocuments.net/reader031/viewer/2022013003/5f075cef7e708231d41c9c66/html5/thumbnails/41.jpg)
• Software interface overheads handicap fast SCM
• Flexible interface is a must for fast SCM
• Aerie: Library file systems help remove generic overheads for higher performance – FlatFS improves performance by up to 110%
41
Conclusion
Thank you! Questions?