Towards Better Understanding of Black-box Auto-Tuning: A ... · 07/13/2018 Towards Better...
Transcript of Towards Better Understanding of Black-box Auto-Tuning: A ... · 07/13/2018 Towards Better...
![Page 1: Towards Better Understanding of Black-box Auto-Tuning: A ... · 07/13/2018 Towards Better Understanding of Black-box Auto-Tuning(ATC’18) 25 Conclusions & Contributions lFirst comparative](https://reader036.fdocuments.net/reader036/viewer/2022062607/6041fd3ee43b7712fe6aa36a/html5/thumbnails/1.jpg)
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 107/13/2018
Towards Better Understanding of Black-box Auto-Tuning: A Comparative
Analysis for Storage Systems
2018 USENIX Annual Technical Conference
Zhen Cao1, Vasily Tarasov2, Sachin Tiwari1, and Erez Zadok1
1Stony Brook University; 2IBM Research – Almaden;
![Page 2: Towards Better Understanding of Black-box Auto-Tuning: A ... · 07/13/2018 Towards Better Understanding of Black-box Auto-Tuning(ATC’18) 25 Conclusions & Contributions lFirst comparative](https://reader036.fdocuments.net/reader036/viewer/2022062607/6041fd3ee43b7712fe6aa36a/html5/thumbnails/2.jpg)
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 207/13/2018
Outlinel Introductionl Backgroundl Experiment Settingsl Evaluationsl Related Workl Conclusions & Future Work
![Page 3: Towards Better Understanding of Black-box Auto-Tuning: A ... · 07/13/2018 Towards Better Understanding of Black-box Auto-Tuning(ATC’18) 25 Conclusions & Contributions lFirst comparative](https://reader036.fdocuments.net/reader036/viewer/2022062607/6041fd3ee43b7712fe6aa36a/html5/thumbnails/3.jpg)
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 307/13/2018
Motivationl Why tuning storage systems?
uSlow storage impacts I/O bound workloadsuDefault settings are sub-optimaluTuning can provide significant gains
§ 9× [FAST’10]
l Manual tuning is intractablel Auto-tuning storage systems
uBlack-box optimization is promisinguLack of comparison of techniquesuLack of understanding
![Page 4: Towards Better Understanding of Black-box Auto-Tuning: A ... · 07/13/2018 Towards Better Understanding of Black-box Auto-Tuning(ATC’18) 25 Conclusions & Contributions lFirst comparative](https://reader036.fdocuments.net/reader036/viewer/2022062607/6041fd3ee43b7712fe6aa36a/html5/thumbnails/4.jpg)
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 407/13/2018
Contributionsl First comparative study on auto-tuning storage
systemsu5 techniques
l Various aspectsuCumulative & instantaneous throughputu Impacts of hyper-parameters
l Explanations on evaluation resultsuFrom storage perspective
l Future Goal: complete solution to tune storage systems
![Page 5: Towards Better Understanding of Black-box Auto-Tuning: A ... · 07/13/2018 Towards Better Understanding of Black-box Auto-Tuning(ATC’18) 25 Conclusions & Contributions lFirst comparative](https://reader036.fdocuments.net/reader036/viewer/2022062607/6041fd3ee43b7712fe6aa36a/html5/thumbnails/5.jpg)
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 507/13/2018
Outlinel Introductionl Backgroundl Experiment Settingsl Evaluationsl Related Workl Conclusions & Future Work
![Page 6: Towards Better Understanding of Black-box Auto-Tuning: A ... · 07/13/2018 Towards Better Understanding of Black-box Auto-Tuning(ATC’18) 25 Conclusions & Contributions lFirst comparative](https://reader036.fdocuments.net/reader036/viewer/2022062607/6041fd3ee43b7712fe6aa36a/html5/thumbnails/6.jpg)
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 607/13/2018
Conceptsl Storage system
u File system, underlying storage hardware and any layers between them
l Parametersu Configurable optionsu E.g., file-system block size
l Parameter valuesu E.g., 1K, 2K, 4K (Ext4 block size)
l Configurationu Combination of parameter values u E.g., [Ext4, 4K, data=ordered]
l Parameter spaceu All possible configurations
l Hyper-parameter
![Page 7: Towards Better Understanding of Black-box Auto-Tuning: A ... · 07/13/2018 Towards Better Understanding of Black-box Auto-Tuning(ATC’18) 25 Conclusions & Contributions lFirst comparative](https://reader036.fdocuments.net/reader036/viewer/2022062607/6041fd3ee43b7712fe6aa36a/html5/thumbnails/7.jpg)
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 707/13/2018
Challengesl Vast parameter space
uExt4: 59 parameters, 1037 configsuDevices, LayersuDistributed, large-scale
l Discrete and non-numericuLinux I/O scheduler: noop, cfq, deadline
l Non-linearity
l Sensitivity to environmentuHardware & workloads
Manual TuningInefficient
GradientUnavailable
![Page 8: Towards Better Understanding of Black-box Auto-Tuning: A ... · 07/13/2018 Towards Better Understanding of Black-box Auto-Tuning(ATC’18) 25 Conclusions & Contributions lFirst comparative](https://reader036.fdocuments.net/reader036/viewer/2022062607/6041fd3ee43b7712fe6aa36a/html5/thumbnails/8.jpg)
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 807/13/2018
Inapplicable Methodsl Control Theory
uUnstable in controlling non-linear systems
l Supervised Machine LearninguLong training phase
uHigh-quality training data
l Inapplicable or inefficient to serve as the core auto-tuning algorithmuCould be helpful as a supplement
![Page 9: Towards Better Understanding of Black-box Auto-Tuning: A ... · 07/13/2018 Towards Better Understanding of Black-box Auto-Tuning(ATC’18) 25 Conclusions & Contributions lFirst comparative](https://reader036.fdocuments.net/reader036/viewer/2022062607/6041fd3ee43b7712fe6aa36a/html5/thumbnails/9.jpg)
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 907/13/2018
Black-box Optimizationl Successfully applied in auto-tuning system
configurations
l ExamplesuGenetic Algorithms (GA)
uSimulated Annealing (SA)
uBayesian Optimization (BO)
l Obliviousness to system’s internals
Configuration EvaluationResults
evaluate
select
![Page 10: Towards Better Understanding of Black-box Auto-Tuning: A ... · 07/13/2018 Towards Better Understanding of Black-box Auto-Tuning(ATC’18) 25 Conclusions & Contributions lFirst comparative](https://reader036.fdocuments.net/reader036/viewer/2022062607/6041fd3ee43b7712fe6aa36a/html5/thumbnails/10.jpg)
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 1007/13/2018
Key Factors
l Fitness: optimization objective(s)
uThroughput, latency, energy, …
uComplex cost functions
l ExplorationuSearch the unvisited area (e.g., randomly)
l ExploitationuUtilize neighborhood or history
l HistoryuHow much past data is kept and used for
exploration/exploitation
![Page 11: Towards Better Understanding of Black-box Auto-Tuning: A ... · 07/13/2018 Towards Better Understanding of Black-box Auto-Tuning(ATC’18) 25 Conclusions & Contributions lFirst comparative](https://reader036.fdocuments.net/reader036/viewer/2022062607/6041fd3ee43b7712fe6aa36a/html5/thumbnails/11.jpg)
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 1107/13/2018
Applied Methodsl Simulated Annealing (SA)
l Genetic Algorithms (GA)
l Deep Q-Network (DQN)
l Bayesian Optimization (BO)
l Random Search (RS)uRandom selection without replacement
![Page 12: Towards Better Understanding of Black-box Auto-Tuning: A ... · 07/13/2018 Towards Better Understanding of Black-box Auto-Tuning(ATC’18) 25 Conclusions & Contributions lFirst comparative](https://reader036.fdocuments.net/reader036/viewer/2022062607/6041fd3ee43b7712fe6aa36a/html5/thumbnails/12.jpg)
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 1207/13/2018
Genetic Algorithmsl Inspired by natural evolutionl Concepts
uGene: file system, block size, …uAllele: Ext4, XFS, Btrfs, …uChromosome: configurationuPopulation: set of configurations
l Selectionl Genetic operators
uCrossoveruMutation
History
Exploitation vs. Exploration
![Page 13: Towards Better Understanding of Black-box Auto-Tuning: A ... · 07/13/2018 Towards Better Understanding of Black-box Auto-Tuning(ATC’18) 25 Conclusions & Contributions lFirst comparative](https://reader036.fdocuments.net/reader036/viewer/2022062607/6041fd3ee43b7712fe6aa36a/html5/thumbnails/13.jpg)
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 1307/13/2018
Outlinel Introductionl Backgroundl Experiment Settingsl Evaluationsl Related Workl Conclusions & Future Work
![Page 14: Towards Better Understanding of Black-box Auto-Tuning: A ... · 07/13/2018 Towards Better Understanding of Black-box Auto-Tuning(ATC’18) 25 Conclusions & Contributions lFirst comparative](https://reader036.fdocuments.net/reader036/viewer/2022062607/6041fd3ee43b7712fe6aa36a/html5/thumbnails/14.jpg)
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 1407/13/2018
Experimental Setupl Hardware
uM1: 2 Intel Xeon single-core 2.8GHz CPU, 2G RAM, 73GB Seagate SCSI drive
uM2: 1 Intel Xeon quad-core 2.4GHz CPU, 24G RAM, 4 drives (SAS-HDD 500GB, SAS-HDD 146GB, 1 SATA-HDD, SSD)
l FilebenchuMacro-workloads: fileserver, mailserver, webserver,
dbserveruDefault working set size
![Page 15: Towards Better Understanding of Black-box Auto-Tuning: A ... · 07/13/2018 Towards Better Understanding of Black-box Auto-Tuning(ATC’18) 25 Conclusions & Contributions lFirst comparative](https://reader036.fdocuments.net/reader036/viewer/2022062607/6041fd3ee43b7712fe6aa36a/html5/thumbnails/15.jpg)
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 1507/13/2018
Experiment Setup (cont.)l Search spaces
uStorage V1§ File system, inode size, block size, block group,
journal options, mount options, special optionsuStorage V2
§ V1 + I/O scheduler§ 6,222 configurations
l MethodologyuExhaustive Search
§ Storage V2: 4 workloads × 4 devices§ 3+ runs for each configuration§ Collected over 2+ years
uSimulate auto-tuning algorithms
![Page 16: Towards Better Understanding of Black-box Auto-Tuning: A ... · 07/13/2018 Towards Better Understanding of Black-box Auto-Tuning(ATC’18) 25 Conclusions & Contributions lFirst comparative](https://reader036.fdocuments.net/reader036/viewer/2022062607/6041fd3ee43b7712fe6aa36a/html5/thumbnails/16.jpg)
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 1607/13/2018
Outline
l Introductionl Backgroundl Experiment Settingsl Evaluationsl Related Workl Conclusions & Future Work
![Page 17: Towards Better Understanding of Black-box Auto-Tuning: A ... · 07/13/2018 Towards Better Understanding of Black-box Auto-Tuning(ATC’18) 25 Conclusions & Contributions lFirst comparative](https://reader036.fdocuments.net/reader036/viewer/2022062607/6041fd3ee43b7712fe6aa36a/html5/thumbnails/17.jpg)
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 1707/13/2018
16
17
18
0 1 2 3 4 5
Bes
tT
hro
ugh
pu
t (k
op
s/s)
Time (hrs)
15.2
18.7
M2-Mailserver-HDD3
GASABO
DQNRS
Best Throughput
![Page 18: Towards Better Understanding of Black-box Auto-Tuning: A ... · 07/13/2018 Towards Better Understanding of Black-box Auto-Tuning(ATC’18) 25 Conclusions & Contributions lFirst comparative](https://reader036.fdocuments.net/reader036/viewer/2022062607/6041fd3ee43b7712fe6aa36a/html5/thumbnails/18.jpg)
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 1807/13/2018
20%
40%
60%
80%
100%
0 1 2 3 4 5
Per
centa
ge
of
Runs
Time (hrs)
M2-Fileserver-HDD3GASABO
DQNRS
Success rate for finding near-optimal configurations
Near-optimal configuration: one with throughput higher than 99% of the global optimal value.
![Page 19: Towards Better Understanding of Black-box Auto-Tuning: A ... · 07/13/2018 Towards Better Understanding of Black-box Auto-Tuning(ATC’18) 25 Conclusions & Contributions lFirst comparative](https://reader036.fdocuments.net/reader036/viewer/2022062607/6041fd3ee43b7712fe6aa36a/html5/thumbnails/19.jpg)
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 1907/13/2018
Instant Throughput
51015
RS
M2-Mailserver-HDD3
51015
SA
M2-Mailserver-HDD3
51015
Thro
ughput
(kops/
s)
GA
M2-Mailserver-HDD3
51015
DQN
M2-Mailserver-HDD351015
0 1 2 3 4 5
BO
Time (hrs)
M2-Mailserver-HDD3
![Page 20: Towards Better Understanding of Black-box Auto-Tuning: A ... · 07/13/2018 Towards Better Understanding of Black-box Auto-Tuning(ATC’18) 25 Conclusions & Contributions lFirst comparative](https://reader036.fdocuments.net/reader036/viewer/2022062607/6041fd3ee43b7712fe6aa36a/html5/thumbnails/20.jpg)
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 2007/13/2018
Genetic Algorithm (GA) Diversity
![Page 21: Towards Better Understanding of Black-box Auto-Tuning: A ... · 07/13/2018 Towards Better Understanding of Black-box Auto-Tuning(ATC’18) 25 Conclusions & Contributions lFirst comparative](https://reader036.fdocuments.net/reader036/viewer/2022062607/6041fd3ee43b7712fe6aa36a/html5/thumbnails/21.jpg)
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 2107/13/2018
l Correlation analysisuOrdinary Least Squares (OLS)uExample: block size and journal option are the
most correlated Ext4 parameter (Fileserver, SSD)
l Explanations on evaluation resultsuGA and BO can identify important parameters
through “history”uSA keeps no “history”; thus performs poorlyuDQN spends too much time on exploration uRandom Search
§ Near-optimal configurations take up 4.5% of the whole search space (M2, Mailserver, HDD).
Correlation Analysis
![Page 22: Towards Better Understanding of Black-box Auto-Tuning: A ... · 07/13/2018 Towards Better Understanding of Black-box Auto-Tuning(ATC’18) 25 Conclusions & Contributions lFirst comparative](https://reader036.fdocuments.net/reader036/viewer/2022062607/6041fd3ee43b7712fe6aa36a/html5/thumbnails/22.jpg)
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 2207/13/2018
Outline
l Introductionl Backgroundl Experiment Settingsl Evaluationsl Related Workl Conclusions & Future Work
![Page 23: Towards Better Understanding of Black-box Auto-Tuning: A ... · 07/13/2018 Towards Better Understanding of Black-box Auto-Tuning(ATC’18) 25 Conclusions & Contributions lFirst comparative](https://reader036.fdocuments.net/reader036/viewer/2022062607/6041fd3ee43b7712fe6aa36a/html5/thumbnails/23.jpg)
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 2307/13/2018
Related Workl Auto-tuning storage
uStorage system design (bin-packing heuristics) [Alvarez et al.]
uData recovery scheduling (GA) [Keeton et al.]uHDF5 optimization (GA) [Behzad et al.]uLustre optimization (DQN) [Li et al.]
l Auto-tuning other systemsuDatabase [Alipourfard et al.]uCloud VMs [Aken et al.]
![Page 24: Towards Better Understanding of Black-box Auto-Tuning: A ... · 07/13/2018 Towards Better Understanding of Black-box Auto-Tuning(ATC’18) 25 Conclusions & Contributions lFirst comparative](https://reader036.fdocuments.net/reader036/viewer/2022062607/6041fd3ee43b7712fe6aa36a/html5/thumbnails/24.jpg)
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 2407/13/2018
Outline
l Introductionl Backgroundl Experiment Settingsl Evaluationsl Related Workl Conclusions & Future Work
![Page 25: Towards Better Understanding of Black-box Auto-Tuning: A ... · 07/13/2018 Towards Better Understanding of Black-box Auto-Tuning(ATC’18) 25 Conclusions & Contributions lFirst comparative](https://reader036.fdocuments.net/reader036/viewer/2022062607/6041fd3ee43b7712fe6aa36a/html5/thumbnails/25.jpg)
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 2507/13/2018
Conclusions & Contributionsl First comparative analysis on 5 techniques on
auto-tuning storage systemsuEfficiency on finding near-optimal configurationsu Instant throughput
l Provide insights from storage perspectiveu Importance of parameters
§ E.g., impact of mutation rates on convergence
l Valuable datasetsuWill release to public
![Page 26: Towards Better Understanding of Black-box Auto-Tuning: A ... · 07/13/2018 Towards Better Understanding of Black-box Auto-Tuning(ATC’18) 25 Conclusions & Contributions lFirst comparative](https://reader036.fdocuments.net/reader036/viewer/2022062607/6041fd3ee43b7712fe6aa36a/html5/thumbnails/26.jpg)
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 2607/13/2018
Future Work
l More complex workloads and search
spaces
l Hyper-parameter tuning
l More sophisticated auto-tuning
u E.g., penalty functions to cope with costly parameter
changes
![Page 27: Towards Better Understanding of Black-box Auto-Tuning: A ... · 07/13/2018 Towards Better Understanding of Black-box Auto-Tuning(ATC’18) 25 Conclusions & Contributions lFirst comparative](https://reader036.fdocuments.net/reader036/viewer/2022062607/6041fd3ee43b7712fe6aa36a/html5/thumbnails/27.jpg)
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 2707/13/2018
Towards Better Understanding of Black-box Auto-Tuning: A Comparative
Analysis for Storage SystemsZhen Cao, Vasily Tarasov, Sachin Tiwari, and Erez Zadok