WiredTiger In-Memory - vs WiredTiger B-Tree...∙ Introducing Percona Memory Engine for MongoDB ∙...
Transcript of WiredTiger In-Memory - vs WiredTiger B-Tree...∙ Introducing Percona Memory Engine for MongoDB ∙...
WiredTiger In-Memoryvs WiredTiger B-Tree
October, 5, 2016 — Movenpick Hotel — AmsterdamSveta Smirnova
∙What is Percona Memory Engine for MongoDB?
∙Typical use cases
∙Advanced Memory Engine
Table of Contents
2
What is Percona Memory Engine for MongoDB?
3
∙ Up to 1000 times faster for OLTP wokloads∙ 10 times faster for read-only workloads∙ Stable throughput∙ No checkpointing
∙ No jitter
Extremely fast In-Memory storage
4
∙ Document-level locking
∙ B-Tree
∙ Practically WiredTiger, but without diskaccess
Based on WiredTiger
5
∙ Doesn’t store data on disk∙ Except small amount of statistics∙ You can control when to log statistics with
option –inMemoryStatisticsLogDelaySecs∙ Still must specify –dbpath
sveta@Thinkie:~/mongo_tests$ ls -lh single/
total 40K
drwxrwxr-x 2 sveta sveta 4,0K Eyl 29 15:00 diagnostic.data
-rw-r--r-- 1 sveta sveta 6 Eyl 29 14:58 mongod.lock
-rw-rw-r-- 1 sveta sveta 93 Eyl 29 14:55 storage.bson
∙ Data does not persist between restarts
WiredTiger without storage
6
∙ –storageEngine=inMemory
∙ Can be only engine on MongoDB server∙ MongoDB restriction, applicable to all engines
∙ Heterogeneous replication and shardingsetups supported
How to enable Memory Engine?
7
∙ Engine can use up to –inMemorySizeGB∙ If data exceeds this amount
∙ WT CACHE FULL error is returned for allkinds of operations that cause user data sizeto grow
INSERTCREATEUPDATE
∙ Reads are not affected
How to control memory usage
8
∙ 100% Open Source
∙ Code available at GitHub
∙ Free for all Percona users and customers
Open Source
9
Typical use cases for Percona Memory Engine
10
∙ Session management∙ Store active sessions in memory∙ Users will receive answer almost immediately∙ Reduce application response time
dramatically
∙ Various temporary collections∙ All you used to store in memcached
Application cache
11
∙ Application runtime data which does notrequire on-disk storage
∙ Intermediary results of calculations
∙ User-specific options
∙ Your idea
Transient Runtime State
12
∙ Thousand-lines aggregations
∙ Temporary collections to store intermediarydata
∙ Complicated queries
Sophisticated data manipulation
13
∙ Large aggregations might be slow∙ Especially if use many collections∙ Often this is not avoidable
∙ To calculate number of distinct values youneed to read whole index
∙ Fast dedicated server is great solution
Real-Time Analytics
14
∙ Data sharing between multi-tier ormulti-language applications
English labels
∙ Articles∙ Pictures∙ Contact
information∙ Other content
Russian labels
Multi-tier object sharing
15
∙ Are you tired to wait when data, needed forapplication test, loads?
∙ Any change in test data causes delay?∙ With Memory engine you can reduce
turnaround time for automated applicationtests.
∙ And still use same syntax
Application Testing
16
Advanced Percona Memory Engine
17
∙ Are you amazed with speed of the Memoryengine?
∙ But still need data to persist betweenrestarts?
∙ You can combine both Memory and WiredTiger in Replica Set or Sharded Cluster
Best of both worlds
18
∙ Setup 2 or more Memory replicas whichcan be Primary
∙ Let WiredTiger to persist data on disk∙ In rare cases if all Memory replicas crash at
the same time you will loose fewtransactions∙ Number of transactions depends on the
latency between In-Memory Primary replicaand WiredTiger replica
Hidden WiredTiger, storing changes in Replica Set
19
Memory
WiredTiger
Memory
Hidden WiredTiger, storing changes in Replica Set
20
rs.initiate(
... {
... "_id" : "rs",
... "members" : [
... {"_id" : 0, "host" : "inMemory1", "priority" : 1},
... {"_id" : 1, "host" : "inMemory2", "priority" : 1},
... {"_id" : 2, "host" : "WiredTiger", "priority" : 0, "hidden" : true}
... ]
... }
)
Hidden WiredTiger to store on disk: example setup
21
∙ Make WiredTiger Primary
∙ Move all reads to read-only Memoryreplicas
∙ Writes will be slow
WiredTiger as Primary in Replica Set
22
Memory
WiredTiger
Memory
WiredTiger as Primary in Replica Set
23
∙ Create Sharded Cluster using Memorynodes only
∙ Split data between nodes
∙ Create copies of data to prevent data loss
Scaling beyond the RAM of a single server
24
Shard 1 Shard 2
Scaling beyond the RAM
25
Shard 1 Shard 2
R1
R2
R3 R4
R5
R6
Scaling beyond the RAM: add redundancy
26
∙ You can use both engines in the ShardedCluster
∙ Split data∙ Session data on Memory nodes∙ Persistent data on WiredTiger node(s)
∙ Duplicate Memory shards to avoid loosingdata
Memory and WiredTiger in Sharded Cluster
27
∙ You can have sharded nodes which useMemory engine
∙ Make them parts of Replica Set
∙ Let hidden WiredTiger member to persistdata on disk
Sharded Cluster Memory and Replica Sets
28
∙ User posts changing rarely are stored ondisk
∙ Session data stored using Memory engine∙ Active comments (last 24 hours) and
actively accessed posts are cached inMemory node
Example: blog application
29
mongos> sh.addShardTag("shard01", "memory") // Memory node
WriteResult( "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 )
mongos> sh.addShardTag("shard02", "memory") // Memory node
WriteResult( "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 )
mongos> sh.addShardTag("shard03", "persist") // WiredTiger node
WriteResult( "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 )
mongos> sh.addShardTag("shard04", "persist") // WiredTiger node
WriteResult( "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 )
Example: tag shards
30
mongos> sh.addTagRange("blog.sessions", { sid: 0 }, { sid: 1000000 }, "memory")
WriteResult({
"nMatched" : 0,
"nUpserted" : 1,
"nModified" : 0,
" id" : {
"ns" : "blog.sessions",
"min" : {
"sid" : 0
}
}
})
Example: split data
31
mongos> sh.addTagRange("blog.comments", { store: "persist", cid: 0 },
... { store: "persist", cid: 1000000 }, "persist")
WriteResult({
...
" id" : {
"ns" : "blog.comments",
"min" : {
"store" : "persist",
"cid" : 0
}}})
mongos> sh.addTagRange("blog.comments", { store: "memory", cid: 0 },
... { store: "memory", cid: 1000000 }, "memory")
WriteResult({
...
Example: split data
31
mongos> sh.addTagRange("blog.posts", { pid: 0 }, { pid: 1000000 }, "persist")
WriteResult({
"nMatched" : 0,
"nUpserted" : 1,
"nModified" : 0,
" id" : {
"ns" : "blog.posts",
"min" : {
"pid" : 0
}
}
})
Example: split data
31
∙ Percona Memory Engine replacesWiredTiger when you need better speedand can afford loosing data
∙ Can be used in setups which combine bothhigh performance of the Memory engineand data persistence of WiredTiger
∙ Open Source
Summary
32
∙ David Bennett
∙ David Murphy
∙ Fernando Ipar
∙ Denis Protyvenskyi
Special thanks
33
∙ Benchmarks for Percona Memory Engine
∙ Introducing Percona Memory Engine forMongoDB
∙ Percona Server for MongoDB manual
∙ Source code for Percona Memory Engine
More informaiton
34
Rate My Session!
35
???
Place for your questions
36
http://www.slideshare.net/SvetaSmirnova
https://twitter.com/svetsmirnova
Thank you!
37