Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million...
Transcript of Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million...
![Page 1: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/1.jpg)
![Page 2: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/2.jpg)
/
/home
/home/A
/home/B
/usr
/usr/A /usr/B
/bin
/bin/A /bin/B
/lib
/lib/A /lib/B
• Strongly consistent metadata.
• Atomic file system operations, such as, move and create
Metadata
![Page 3: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/3.jpg)
![Page 4: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/4.jpg)
• {operation} [flags] {path(s)}•
![Page 5: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/5.jpg)
•
•
![Page 6: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/6.jpg)
•
•
•
![Page 7: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/7.jpg)
•
•
•
•
![Page 8: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/8.jpg)
•
•
![Page 9: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/9.jpg)
![Page 10: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/10.jpg)
→ → →
→ →
→ → →
→
→ → →
→ →
![Page 11: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/11.jpg)
![Page 12: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/12.jpg)
•
•
•
![Page 13: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/13.jpg)
→ → →
→ → →
→
→ → →
→ →
→ →
![Page 14: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/14.jpg)
![Page 16: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/16.jpg)
![Page 17: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/17.jpg)
![Page 18: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/18.jpg)
![Page 19: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/19.jpg)
create file /user/F4 create file /user/F4
![Page 20: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/20.jpg)
●
●
●
●
●
●
●
●
![Page 21: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/21.jpg)
![Page 22: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/22.jpg)
●
●
●
●
●
●
●
![Page 23: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/23.jpg)
![Page 24: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/24.jpg)
![Page 25: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/25.jpg)
![Page 26: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/26.jpg)
Select * from Inodes where PID = 1
drwxrwx--- /home/Adrwxrwx--- /home/B
![Page 27: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/27.jpg)
Select * from Inodes where PID = 1
WITH(INDEX(…))
drwxrwx--- /home/Adrwxrwx--- /home/B
![Page 28: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/28.jpg)
•
![Page 29: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/29.jpg)
Hash Fn
PID % 4 = Partition No
![Page 30: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/30.jpg)
Select * from Inodes where PID = 2
WITH(INDEX(…))
drwxrwx--- /home/Adrwxrwx--- /home/B
![Page 31: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/31.jpg)
Start Transaction on Node 2Select * from Inodes where PID = 2
WITH(INDEX(name))
drwxrwx--- /home/Adrwxrwx--- /home/B
![Page 32: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/32.jpg)
••
•
•
•
•
•
![Page 33: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/33.jpg)
•
••
•
•
•
![Page 34: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/34.jpg)
![Page 35: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/35.jpg)
![Page 36: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/36.jpg)
![Page 37: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/37.jpg)
•
•
•
•
• Identical avg op latency (~3ms) for small number (50) of clients
• 10X lower latency for large number (6500) of clients
•
• 37X more metadata.
![Page 38: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/38.jpg)
![Page 39: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/39.jpg)
![Page 40: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/40.jpg)
![Page 41: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/41.jpg)
![Page 42: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/42.jpg)
Block Storage Layer for Small and Large Files
Metadata Storage
Small Files Storage
Block Storage Layer for Large Files
![Page 43: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/43.jpg)
![Page 44: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/44.jpg)
![Page 46: Strongly consistent metadata. - GitHub Pages · 2020. 9. 6. · Scaling HDFS to more than 1 million operations per second with HopsFS M Ismail, S Niazi, M Ronström, S Haridi, J Dowling](https://reader034.fdocuments.net/reader034/viewer/2022051917/6009ca556ff5d010a61625ac/html5/thumbnails/46.jpg)
Read More
Scaling Distributed Hierarchical File Systems Using NewSQL Databases
Salman Niazi. Ph.D. Thesis. KTH Royal Institute of Technology
Scaling hierarchical file system metadata using newsql databases
S Niazi, M Ismail, S Haridi, J Dowling, S Grohsschmiedt, M Ronström
15th USENIX Conference on File and Storage Technologies (FAST 17), 89-104
Scaling HDFS to more than 1 million operations per second with HopsFS
M Ismail, S Niazi, M Ronström, S Haridi, J Dowling
2017 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid …
Size Matters: Improving the Performance of Small Files in Hadoop
S Niazi, M Ronström, S Haridi, J Dowling
Proceedings of the 19th International Middleware Conference, 26-39