PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with...
-
Upload
virgil-powers -
Category
Documents
-
view
233 -
download
0
Transcript of PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with...
PrincetonUniversity
From application requests to Virtual IOPs:Provisioned key-value storage with Libra
David Shue* and Michael J. Freedman
(*now at Google)
Shared Cloud
Tenant A Tenant BVM VM VM VM VM VM
Tenant CVM VM VM
Shared Cloud Storage
Tenant A Tenant BVM VM VM VM VM VM
Key-Value Storage
Block Storage
SQL Database
Tenant CVM VM VM
Unpredictable Shared Cloud Storage
Tenant A Tenant BVM VM VM VM VM VM
Key-Value Storage
Block Storage
SQL Database
Tenant CVM VM VM
Disk IO-bound TenantsSSD-backed storage
Key-Value Storage
Provisioned Shared Key-Value Storage
Tenant AVM VM VM
Tenant BVM VM VM
Tenant CVM VM VM
Application Requests
Shared Key-Value Storage
Low-level IO
SSDSSD SSD SSD SSD
GET/sPUT/s
IOPsBW
ReservationA ReservationB ReservationC
(1KB normalized)
6
Libra Contributions
•Libra IO Scheduler- Provisions low-level IO allocations for app-
request reservations w/ high utilization.
- Supports arbitrary object distributions and workloads.
•2 key mechanisms- Track per-tenant app-request resource profiles.
- Model IO resources with Virtual IOPs.
7
Related Work
Storage Type
App-reques
ts
WorkConservin
g
Media
Maestro Block N N HDD
mClock Block N Y HDD
FlashFQ Block N Y SSD
DynamoDB
Key-Value
Y N SSD
Provisioned Distributed Key-Value Storage
Tenant A Tenant BVM VM VM VM VM VM
ReservationA ReservationB Tenant B
VM VM VM
Storage Node N
Storage Node 1 ...Shared Key-Value Storage
Global Reservation Problem
Local Reservation Problem
[Pisces OSD1 ’12]
9
Storage Node N
Provisioned Distributed Key-Value Storage
ReservationA ReservationB
Storage Node 1
data partitionsdem
an
d
...
Tenant AVM VM VM
Tenant BVM VM VM
10
Storage Node N
Storage Node 1
Provisioned Distributed Key-Value Storage
ReservationA ReservationB
...
ReservationA ReservationB
11
Storage Node N
Storage Node 1
Provisioned Distributed Key-Value Storage
...
ResAn ResBnResA1 ResB1 ReservationA ReservationB
ReservationA ReservationB
12
Key-Value Protocol
Persistence Engine
IO Scheduler
Physical Disk
Retrieve K
Read l337
IO operation
Provisioned Local Key-Value Storage
GET K
GET 1001100
Libra IO Scheduler
13
blah
blah
Libra Design
LibraProvisionin
gPolicy
Libra IO Scheduler
GET
PU
T
How much IO to consume?
Reservation
Distribution Policy
How much IO to provision?
Persistence Engine
Physical Disk
DRR
14
Provisioning App-request Reservations is Hard
IO Amplification
IO Interference
Non-linear IO Performance
1 KB PUT ≥ 1 KB Write
Variable IO throughput
Underestimate provisionable IO
Non-linear cost per KB
Model IO with Virtual IOPs
Track tenant app-request resource profiles
15
Workload-dependent IO AmplificationPU
T PUT K,V3
V3V2
FLU
SH
V3 indexA M
LevelDB (LSM-Tree)
CO
MPA
CT
V1 indexH V
V0 indexF O
V3 index
A O
16
Workload-dependent IO AmplificationPU
TFL
US
HC
OM
PA
CT
PUT K,V3
V3V2
V3 indexA M
V1 indexH V
V0 indexF O
V3 index
A O
LevelDB (LSM-Tree)
17
Workload-dependent IO AmplificationPU
TFL
US
HC
OM
PA
CT
PUT K,V3
V3V2
V3 indexA M
V1 indexH V
V0 indexF O
V3 index
A O
LevelDB (LSM-Tree)
18
Workload-dependent IO Amplification
GET K
indexA M
indexH V
indexF O
K indexG Z
19
Libra Tracks App-request IO Consumption to Determine IO
Allocations
5 GET
25 PUT
FLUSH
COMPACT
Per-PUT
Per-GET
Compute app-request IO
profiles
GET
x
x PU
T IO=
Tenant A 5 IO units
PUT
FLUSH
Track IOconsumption
Provision IOallocations
blah
blah
LibraProvisionin
gPolicy
IOTenant A 500 IO/s
5
100
50
1
+6
10.5
80
70 500
20
Unpredictable IO Interference
Die-level parallelism, low latency IOPs
4 read/4 write tenants
Shared-controller and bus contention
Erase-before-write overhead
FTL and read-modify-write garbage colleciton
21
Unpredictable IO Interference
21
22
Libra Underestimates IO Capacity to Ensure Provisionable Throughput
Provisionable IO throughput = floor(workloads)
(18 Kop/s)Pro
vis
ionable
IO
lim
it
23
Libra Underestimates IO Capacity to Ensure Provisionable Throughput
Provisionable IO throughput = floor(workloads)
(18 Kop/s)Pro
vis
ionable
IO
lim
it
24
Libra Underestimates IO Capacity to Ensure Provisionable Throughput
Provisionable IO throughput = floor(workloads)
(18 Kop/s)Pro
vis
ionable
IO
lim
it
25
Non-linear IO Performance
Max BW Max IOP/s
26
Non-linear IO Performance
Max BW Max IOP/s
27
Libra Uses Virtual IOPs to Model IO Resources
Unifies IO cost into a single metric
Captures non-linear IO performance
Provides IO insulation
2 equal-allocation tenantsIO Insulation = 1/2 Max
Read/Write
28
Libra Uses Virtual IOPs to Model IO Resources
2 equal-allocation tenantsIO Insulation = 1/2 Max
Read/Write
29
blah
blah
Libra Design
LibraProvisionin
gPolicy
Libra IO Scheduler
Persistence Engine
Physical Disk
Charge tenant IOPs based on
VOP cost
Track app-request VOP consumption
Provision VOPs within
provisionable limit
Update tenant VOP
allocations
30
Evaluation
• Does Libra's IO resource model achieve accurate resource allocations?
• Does Libra's IO threshold make an acceptable tradeoff of performance for predictability in a real storage stack?
• Can Libra ensure per-tenant app-request reservations while achieving high utilization?
31
Interference-free Ideal
Libra Achieves Accurate IO Allocations
Throughput Ratio = Actual / Expected (IO Insulation)
Read 1 KB
even
32
Interference-free Ideal
Libra Achieves Accurate IO Allocations
Write 1-256 KB
Throughput Ratio = Actual / Expected (IO Insulation)
33
Libra Achieves Accurate IO Allocations
34
Libra Achieves Accurate IO Allocations
35
Libra Achieves Accurate IO AllocationsMin-Max Ratio =
Min Throughput Ratio / Max Throughput Ratio
36
Libra Trades-off Nominal IO Throughput For Predictability
37
Libra Trades-off Nominal IO Throughput For Predictability
< 10th percentile covered by SLA and higher-level policies
PercentileWorkloa
d10th 50th 80th All
99:1 1.6% 30.5% 40.5% 45.8%
25:75 1.4% 14.9% 25.0% 34.7%
1:99 0.7% 12.2% 19.5% 28.1%
Unprovisionable Throughput As a Percentage of Total Throughput
38
Libra Achieves App-request Reservations
Read Heavy Mixed Write Heavy
1.5x0.5x
Work-conserving
consumption of unprovisioned
resources
Fully provisioned allocations
39
Libra Achieves App-request Reservations
Read Heavy Mixed Write Heavy
40
•Libra IO Scheduler- Provisions IO allocations for app-request
reservations w/ high utilization.
- Supports arbitrary object distributions and workloads.
•2 key mechanisms
- Track per-tenant app-request resource profiles.
- Model IO resources with Virtual IOPs.
•Evaluation
- Achieves accurate low-level IO allocations.
- Provisions the majority of IO resources over a wide range of workloads
- Satisfies app-request reservations w/ high utilization.
Conclusion