1 Directed Acyclic Graph DAG – directed graph with no directed cycles.
UniStore Project Updates...• Overview of a pattern-directed replication scheme • Object distance...
Transcript of UniStore Project Updates...• Overview of a pattern-directed replication scheme • Object distance...
The Center for Cloud and Autonomic Computing is supported by the National Science Foundation under Grant No. 1362134.
NSF CAC Semi-Annual Meeting, April 28-29, 2016
UniStore Project UpdatesWei Xie, Jiang Zhou, Mark Reyes, Jason Nobel and Yong ChenDepartment of Computer Science and Nimboxx, Inc.
Abstract• 2 new papers completed
• Tiered-CRUSH• Pattern-directed Replication Scheme
• 1 new paper in preparation: version consistent hash• 2 papers under review: PRS and SUORA• Simulation code and prototype developed to evaluate
the proposed schemes• Proposed schemes improve the performance of
heterogeneous storage systems and maintain the balanced storage utilization
• Most data centers use heterogeneous storage combining hard disk drives HDD with emergingstorage class memory SCM (e.g., solid state drives SSD and phase change memory PCM)
• Lack a storage system that unifies the management of heterogeneous storage devices efficiently
• Data are replicated for data availability, but not for performance
• To take full advantage of faster storage devices’ performance and the hard disk drives’ cost-efficiency
• Overview of a pattern-directed replication scheme
• Object distance calculation
• Evaluation
Motivation and Goals
Research Paper 2: Pattern-directed Replication Scheme
AcknowledgementsWe are grateful to the Cloud and Autonomic Computing site at Texas Tech University for the valuable support for this project. Thank High Performance Computing Center at Texas Tech University for providing the computing resources and support for this project.
• CRUSH ensures data placed across multiple independent locations to improve data availability
• Tiered-CRUSH integrates storage tiering into the CRUSH data placement
• The virtualized volumes have different access pattern• Access frequency of object recorded per volume, hotter
data more likely to be placed on faster tiers• Fair storage utilization maintained
(1) The original objects are placed on hybrid nodes in default layout(2) The runtime object I/O requests are traced by a trace collector(3) The scheme analyzes the trace, reorganizes objects for identified access patterns and makes replications(4) when applications run again, objects are read from replicas in optimized layout
(a) Random Read (c) Overhead
Real machine/Virtual machineApplication’s
execution
Default layout
Object I/O request
Heterogeneous Nodes
Trace collector
I/O trace
Trace analyzer
Real machine/Virtual machineApplication’s
execution
Optimized layout
Object I/O redirection
Heterogeneous Nodes
Distribution algorithm
Objectreorganization
Data replicator
Pattern-directed Replication Scheme (PRS)
(a) Local access pattern
(b) Global access pattern
dist(o1,o2 ) =1−Mincount(o1,o2 )count(o1)
, count(o1,o2 )count(o2 )
"#$
%&'
0
0
Object ID 1 2 3 4 5 6 7 8 9 10
2 4 6New Object 7 9 10
Replica 1 Replica 2
local pattern local pattern
dist(o1,o2 ) =count(o1)− count(o2 )count(o1)+ count(o2 )
"#$
%&'
HDD
0
0
Object ID 1 2 3 4 5 50 51 52 53
2 5 1 4New Object 3
Replica 1 Replica 2
Hot data
50
Replica 3
5351
SCM Heterogeneous device architecture
global pattern global pattern global pattern(ratio = SCM capacity
Total capacity)
0
20
40
60
80
100
120
140
1 2 4 8 16 32 64 128 256 512
Agg
rega
ted
Ban
dwid
th (M
B/s)
Number of processes
FIO with different numbers of processesOriginal-1 rep Original-3 rep PD only
0
20
40
60
80
100
120
140
1 2 4 8 16 32 64 128 256 512A
ggre
gate
d B
andw
idth
(MB/
s)
Number of processes
FIO with different numbers of processesOriginal-1 rep Original-3 rep PD only
(b) Sequential ReadResearch Paper 1: Tiered-CRUSH
0
20
40
60
80
100
120
140
1 2 4 8 16 32 64 128 256 512
Agg
rega
ted
Ban
dwid
th (M
B/s)
Number of processes
FIO with different numbers of processesOriginal-1 rep PRS scheme
Root
Tier1 Tier2 Tier3
N3 S1 S2 H1 H2 H3
Root
Cab1 Cab2 Cab3
N3S1 S2H1 H2 H3
SATA SSD: S NVMe SSD: N Hard Disk Drive: H
Primary Replica1 Replica2
Primary placed based on tiering information and tier map
Replicas placed based on CRUSH map
Data Object
Tier Map CRUSH Map
Volume 1 Volume 2
Storage tiers
Objects Read count Objects Write count
Hotness Server
Monitor accessCalculate and
maintain tiering information
Tier1 Tier2 Tier3
• Build versions into the virtual nodes• Avoid data migration when adding nodes or node fails• Maintain efficient data lookup • Example of 3 versions of a consistent hashing ring
• Data lookup algorithm
Research Paper 3: Version Consistent Hashing
21 3
Version1, committed
1
2
3c
c
cD1
D1Update, lookup
21 3
Version2, uncommitted
1
2
3c
c
cD1
D1
4
4n, v2
Update, lookupLookup
21 3
Version3, uncommitted
1
2
3c
c
cD1
D1
4
4n, v2
Update, lookupLookup
5n, v3
5
lookup
21 3
Version3, committed
1
2
3c
c
cD1
D1
4
5 c
Update, lookup
4 c
5
D1
3
1
2
4
5
6
7
v2
v3
v3
v4
• v1: 1,2• v2: 4,1• v3: 4,6• v4: 4,6• Lookup locations: {4, 6, 1, 2}
• Performance Improvement
0 500 1000 1500 2000Number of nodes (half are added)
100
101
102
103
104
Aver
age
look
up a
mpl
ifica
tion Consistent Hashing
Commit CHVersion CH