Dirty - How simple is your database?
-
Upload
felix-geisendoerfer -
Category
Technology
-
view
8.149 -
download
1
description
Transcript of Dirty - How simple is your database?
![Page 1: Dirty - How simple is your database?](https://reader034.fdocuments.net/reader034/viewer/2022042714/54b794d34a79591d4a8b46bc/html5/thumbnails/1.jpg)
Dirty NoSQLHow simple is your database?
25.09.2010
![Page 2: Dirty - How simple is your database?](https://reader034.fdocuments.net/reader034/viewer/2022042714/54b794d34a79591d4a8b46bc/html5/thumbnails/2.jpg)
About
Contributor Co-founder
node.js driver node.js driver
Felix Geisendörfer
![Page 3: Dirty - How simple is your database?](https://reader034.fdocuments.net/reader034/viewer/2022042714/54b794d34a79591d4a8b46bc/html5/thumbnails/3.jpg)
![Page 4: Dirty - How simple is your database?](https://reader034.fdocuments.net/reader034/viewer/2022042714/54b794d34a79591d4a8b46bc/html5/thumbnails/4.jpg)
Dirty
![Page 5: Dirty - How simple is your database?](https://reader034.fdocuments.net/reader034/viewer/2022042714/54b794d34a79591d4a8b46bc/html5/thumbnails/5.jpg)
Dirty
Dirty
JavaScript views In-memory
![Page 6: Dirty - How simple is your database?](https://reader034.fdocuments.net/reader034/viewer/2022042714/54b794d34a79591d4a8b46bc/html5/thumbnails/6.jpg)
Design choices
![Page 7: Dirty - How simple is your database?](https://reader034.fdocuments.net/reader034/viewer/2022042714/54b794d34a79591d4a8b46bc/html5/thumbnails/7.jpg)
Non-blocking I/O FTW
![Page 8: Dirty - How simple is your database?](https://reader034.fdocuments.net/reader034/viewer/2022042714/54b794d34a79591d4a8b46bc/html5/thumbnails/8.jpg)
150 lines of code
Dirty
CouchDB
Redis
0 5.000 10.000 15.000 20.000
Lines of code
![Page 9: Dirty - How simple is your database?](https://reader034.fdocuments.net/reader034/viewer/2022042714/54b794d34a79591d4a8b46bc/html5/thumbnails/9.jpg)
Append-only JSON log
$ cat dirty.db
{"key": "first", "val": {"foo": "bar"}}{"key": "second", "val": "A string"}{"key": "a-number", "val": 23}
![Page 10: Dirty - How simple is your database?](https://reader034.fdocuments.net/reader034/viewer/2022042714/54b794d34a79591d4a8b46bc/html5/thumbnails/10.jpg)
No Networking
var db = require('dirty')('test.db');
db.set('foo', 'bar');db.get('foo'); // => bar
![Page 11: Dirty - How simple is your database?](https://reader034.fdocuments.net/reader034/viewer/2022042714/54b794d34a79591d4a8b46bc/html5/thumbnails/11.jpg)
No Networking
var db = require('dirty')('languages.db');
db.set('javascript', {age: 15});db.set('python', {age: 19});db.set('perl', {age: 23});
db.forEach(function(key, doc) { console.log('%s is %d years old.', key, doc.age);});
![Page 12: Dirty - How simple is your database?](https://reader034.fdocuments.net/reader034/viewer/2022042714/54b794d34a79591d4a8b46bc/html5/thumbnails/12.jpg)
Performance
![Page 13: Dirty - How simple is your database?](https://reader034.fdocuments.net/reader034/viewer/2022042714/54b794d34a79591d4a8b46bc/html5/thumbnails/13.jpg)
Benchmarks
Do your own!
![Page 14: Dirty - How simple is your database?](https://reader034.fdocuments.net/reader034/viewer/2022042714/54b794d34a79591d4a8b46bc/html5/thumbnails/14.jpg)
dirty.get()
50 MHz(50 million / s)
v8: 160 MHz
![Page 15: Dirty - How simple is your database?](https://reader034.fdocuments.net/reader034/viewer/2022042714/54b794d34a79591d4a8b46bc/html5/thumbnails/15.jpg)
dirty.set()
5 MHz(5 million / s)
v8: 12 MHz
![Page 16: Dirty - How simple is your database?](https://reader034.fdocuments.net/reader034/viewer/2022042714/54b794d34a79591d4a8b46bc/html5/thumbnails/16.jpg)
dirty.set()
200 kHz(200-thousand / s)
Numbers
70 kHz(70-thousand / s)
256 byte string
With flushing to disk
![Page 17: Dirty - How simple is your database?](https://reader034.fdocuments.net/reader034/viewer/2022042714/54b794d34a79591d4a8b46bc/html5/thumbnails/17.jpg)
dirty.forEach()
4.5 MHz(4.5 million / s)
v8: 33 MHz
![Page 18: Dirty - How simple is your database?](https://reader034.fdocuments.net/reader034/viewer/2022042714/54b794d34a79591d4a8b46bc/html5/thumbnails/18.jpg)
The Wall
• Dirty is a wonderful database as long as you have < 1 million records
• After that, you hit “The Wall”
![Page 19: Dirty - How simple is your database?](https://reader034.fdocuments.net/reader034/viewer/2022042714/54b794d34a79591d4a8b46bc/html5/thumbnails/19.jpg)
Scaling beyondPossibilities for future node.js based databases
![Page 20: Dirty - How simple is your database?](https://reader034.fdocuments.net/reader034/viewer/2022042714/54b794d34a79591d4a8b46bc/html5/thumbnails/20.jpg)
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
22.000
0
2000
4000
6000
8000
10.000
12.000
14.000
16.000
18.000
20.000
Year
PB /
mon
th
2000 20022003
2004
2005
2006
2007
2008
2009
2010
The Internet
![Page 21: Dirty - How simple is your database?](https://reader034.fdocuments.net/reader034/viewer/2022042714/54b794d34a79591d4a8b46bc/html5/thumbnails/21.jpg)
22 exabyte / month(that is 22 billion gigabytes)
![Page 22: Dirty - How simple is your database?](https://reader034.fdocuments.net/reader034/viewer/2022042714/54b794d34a79591d4a8b46bc/html5/thumbnails/22.jpg)
Flexible guarantees
db.set('my-key', 'my-value', function(err) { if (err) throw err;
console.log('Record written to disk.');});
console.log('Record written to memory.');
![Page 23: Dirty - How simple is your database?](https://reader034.fdocuments.net/reader034/viewer/2022042714/54b794d34a79591d4a8b46bc/html5/thumbnails/23.jpg)
Memory / Disk Hybrids
• “Memcached” built into your database
• Better knowledge about your data than any general purpose algorithm
![Page 24: Dirty - How simple is your database?](https://reader034.fdocuments.net/reader034/viewer/2022042714/54b794d34a79591d4a8b46bc/html5/thumbnails/24.jpg)
Replication
• Node.js = perfect for streaming between instances
• Node could “hold” the connection if a not-yet replicated key is being requested
![Page 25: Dirty - How simple is your database?](https://reader034.fdocuments.net/reader034/viewer/2022042714/54b794d34a79591d4a8b46bc/html5/thumbnails/25.jpg)
Web Services
• Node.js could act as a proxy for different database backends (written in node or not)
• One could query 3rd party services for information
![Page 27: Dirty - How simple is your database?](https://reader034.fdocuments.net/reader034/viewer/2022042714/54b794d34a79591d4a8b46bc/html5/thumbnails/27.jpg)
Get it
http://github.com/felixge/node-dirty
$ npm install dirty
or
![Page 28: Dirty - How simple is your database?](https://reader034.fdocuments.net/reader034/viewer/2022042714/54b794d34a79591d4a8b46bc/html5/thumbnails/28.jpg)
![Page 29: Dirty - How simple is your database?](https://reader034.fdocuments.net/reader034/viewer/2022042714/54b794d34a79591d4a8b46bc/html5/thumbnails/29.jpg)
Don’t hit the wall
![Page 31: Dirty - How simple is your database?](https://reader034.fdocuments.net/reader034/viewer/2022042714/54b794d34a79591d4a8b46bc/html5/thumbnails/31.jpg)
Sources
• Internet traffic numbers: http://en.wikipedia.org/wiki/Internet_traffic
• Wonderful dino-rider vs. volcano image: Jesse Star / http://www.geekologie.com/2009/10/dinorider_geekologie_writer_vs.php
![Page 32: Dirty - How simple is your database?](https://reader034.fdocuments.net/reader034/viewer/2022042714/54b794d34a79591d4a8b46bc/html5/thumbnails/32.jpg)
Memory overhead
• 20mb overhead / 1 million records
• 20 bytes / key
(for setting numeric key / values)