The Hive Think Tank: Rocking the Database World with RocksDB
-
Upload
the-hive -
Category
Technology
-
view
593 -
download
2
Transcript of The Hive Think Tank: Rocking the Database World with RocksDB
![Page 1: The Hive Think Tank: Rocking the Database World with RocksDB](https://reader034.fdocuments.net/reader034/viewer/2022042600/587146881a28ab55588b592b/html5/thumbnails/1.jpg)
RocksDB
Dhruba BorthakurFeb 3, 2016.
Open Discussion on Roadmap and Vision
![Page 2: The Hive Think Tank: Rocking the Database World with RocksDB](https://reader034.fdocuments.net/reader034/viewer/2022042600/587146881a28ab55588b592b/html5/thumbnails/2.jpg)
How many of you have hands-on experience with RocksDB?Use RocksDB?
Contribute to RocksDB?
![Page 3: The Hive Think Tank: Rocking the Database World with RocksDB](https://reader034.fdocuments.net/reader034/viewer/2022042600/587146881a28ab55588b592b/html5/thumbnails/3.jpg)
RocksDB is an open source-projectYou get to decide and contribute how to take it forward
![Page 4: The Hive Think Tank: Rocking the Database World with RocksDB](https://reader034.fdocuments.net/reader034/viewer/2022042600/587146881a28ab55588b592b/html5/thumbnails/4.jpg)
What is RocksDB?
•Key-Value persistent store•Embedded•Optimized for fast storage•Server workloads
![Page 5: The Hive Think Tank: Rocking the Database World with RocksDB](https://reader034.fdocuments.net/reader034/viewer/2022042600/587146881a28ab55588b592b/html5/thumbnails/5.jpg)
RocksDB API▪Keys and values are arbitrary byte arrays.
▪Data are stored sorted by key.
▪Update Operations: Put/Delete/Merge
▪Queries: Get/Iterator
![Page 6: The Hive Think Tank: Rocking the Database World with RocksDB](https://reader034.fdocuments.net/reader034/viewer/2022042600/587146881a28ab55588b592b/html5/thumbnails/6.jpg)
RocksDB Architecture
Write Request
Read Request
FlushCompaction
Active MemTable
ReadOnlyMemTable
log
log
log
LSM
sst sst
sst sst
sst
sst
d
Switch
Memory Persistent Storage
Switch
![Page 7: The Hive Think Tank: Rocking the Database World with RocksDB](https://reader034.fdocuments.net/reader034/viewer/2022042600/587146881a28ab55588b592b/html5/thumbnails/7.jpg)
Why does FB develop RocksDB?
![Page 8: The Hive Think Tank: Rocking the Database World with RocksDB](https://reader034.fdocuments.net/reader034/viewer/2022042600/587146881a28ab55588b592b/html5/thumbnails/8.jpg)
RocksDB in the mainstream
•Reduces a 5 TB MongoDB instance to 285 GB on MongoRocks
![Page 9: The Hive Think Tank: Rocking the Database World with RocksDB](https://reader034.fdocuments.net/reader034/viewer/2022042600/587146881a28ab55588b592b/html5/thumbnails/9.jpg)
MySQL Innodb vs RocksDB
•LinkBench: open source benchmark for Facebook’s workload•Reduces MySQL flash storage space by 50% for LinkBench
![Page 10: The Hive Think Tank: Rocking the Database World with RocksDB](https://reader034.fdocuments.net/reader034/viewer/2022042600/587146881a28ab55588b592b/html5/thumbnails/10.jpg)
It is easy to use RocksDB in your software
![Page 11: The Hive Think Tank: Rocking the Database World with RocksDB](https://reader034.fdocuments.net/reader034/viewer/2022042600/587146881a28ab55588b592b/html5/thumbnails/11.jpg)
RocksDB: Open & Pluggable
Pluggable Memtable format
in RAM
Write Request from ApplicationGet or Scan Request from Application
Transaction log Pluggable sst data format on storage
Pluggable Compaction
Blooms
CustomizableWAL
![Page 12: The Hive Think Tank: Rocking the Database World with RocksDB](https://reader034.fdocuments.net/reader034/viewer/2022042600/587146881a28ab55588b592b/html5/thumbnails/12.jpg)
RocksDB is a tool, not a solution by itself.Embed it into your software solution.
![Page 13: The Hive Think Tank: Rocking the Database World with RocksDB](https://reader034.fdocuments.net/reader034/viewer/2022042600/587146881a28ab55588b592b/html5/thumbnails/13.jpg)
RocksDB Integrations
•LinkedIn Feed , Yahoo Sherpa, Airbnb, Pinterest•Microsoft Bing Platform (development)•Apache Samza•Rippled server (bitcoin style)•RedHat CEPH•Open Channel SSD
![Page 14: The Hive Think Tank: Rocking the Database World with RocksDB](https://reader034.fdocuments.net/reader034/viewer/2022042600/587146881a28ab55588b592b/html5/thumbnails/14.jpg)
Any SSD Vendors out here today?Make RocksDB use your SSD efficiently!
![Page 15: The Hive Think Tank: Rocking the Database World with RocksDB](https://reader034.fdocuments.net/reader034/viewer/2022042600/587146881a28ab55588b592b/html5/thumbnails/15.jpg)
Any FileSystem vendors here today?Make RocksDB use your filesystem efficiently!
![Page 16: The Hive Think Tank: Rocking the Database World with RocksDB](https://reader034.fdocuments.net/reader034/viewer/2022042600/587146881a28ab55588b592b/html5/thumbnails/16.jpg)
Want to extend RocksDB functionality?Use StackableDB
e.g. TTL Support, Geo-index, Redis-style
![Page 17: The Hive Think Tank: Rocking the Database World with RocksDB](https://reader034.fdocuments.net/reader034/viewer/2022042600/587146881a28ab55588b592b/html5/thumbnails/17.jpg)
Vision for the future
•Most performant database engine on ram, SSD and disks• Optimize for next-generation storage hardware
•Flexibility to be deployed on varied environments• RocksDB’s components are pluggable• Enables software vendors to customize their solution
![Page 18: The Hive Think Tank: Rocking the Database World with RocksDB](https://reader034.fdocuments.net/reader034/viewer/2022042600/587146881a28ab55588b592b/html5/thumbnails/18.jpg)
Roadmap for 2016
What do you want from RocksDB?
Here are some probable enhancements for RocksDB in 2016
![Page 19: The Hive Think Tank: Rocking the Database World with RocksDB](https://reader034.fdocuments.net/reader034/viewer/2022042600/587146881a28ab55588b592b/html5/thumbnails/19.jpg)
Raw Devices
RocksDB would run on a device lun
RocksDB could use directIO and bypass OS cache
![Page 20: The Hive Think Tank: Rocking the Database World with RocksDB](https://reader034.fdocuments.net/reader034/viewer/2022042600/587146881a28ab55588b592b/html5/thumbnails/20.jpg)
MergeOperator & CompactionFilter
RocksDB could support a lua-based Filter
RocksDB could support a Javascript-based Filter
![Page 21: The Hive Think Tank: Rocking the Database World with RocksDB](https://reader034.fdocuments.net/reader034/viewer/2022042600/587146881a28ab55588b592b/html5/thumbnails/21.jpg)
Reduce storage footprint
Dictionary based compression
More tradeoffs between cpu and compression (zstd, etc)
![Page 22: The Hive Think Tank: Rocking the Database World with RocksDB](https://reader034.fdocuments.net/reader034/viewer/2022042600/587146881a28ab55588b592b/html5/thumbnails/22.jpg)
Production support
Database repair
Standalone utilities to inspect files, blocks, MANIFEST
![Page 23: The Hive Think Tank: Rocking the Database World with RocksDB](https://reader034.fdocuments.net/reader034/viewer/2022042600/587146881a28ab55588b592b/html5/thumbnails/23.jpg)
Tiered Storage
Single db can have RAM, SSD, disk & next-gen-hardware
Intelligent movement of data between storage tiers
![Page 24: The Hive Think Tank: Rocking the Database World with RocksDB](https://reader034.fdocuments.net/reader034/viewer/2022042600/587146881a28ab55588b592b/html5/thumbnails/24.jpg)
Open Discussion