Memcached + Innodb = Performance. The Waffle Grid Project.
-
Upload
oleksiy-kovyrin -
Category
Documents
-
view
840 -
download
0
description
Transcript of Memcached + Innodb = Performance. The Waffle Grid Project.
Partners.org | © MySQL AB 2005 | www.mysql.com 1
Memcached + Innodb = Performance
The Waffle Grid Project
Matt Yonkovit and Yves Trudeau
2Copyright 2009 MySQL AB The World’s Most Popular Open Source Database
Agenda
• Why?• Presenting WaffleGrid• Benchmarks• Status and roadmap• WaffleGrid quick start• Questions
3Copyright 2009 MySQL AB The World’s Most Popular Open Source Database
Why?: Common Problems
• Clients Face Similar issues – Some clients are stuck with 32 bit architectures– Other clients actually exceed their current servers
resources– While other clients simply outgrow their disk before they
outgrow the rest of their server– Many clients don't have the financial resources to scrap
their “good enough” servers to buy servers with > 128GB of memory
4Copyright 2009 MySQL AB The World’s Most Popular Open Source Database
Key Goal's for High Performance
• As much “hot” ( heavily used ) data should fit into memory as possible ( all is best )
– Disk IO is the one of if not the most expensive operation in your system
– Millisecond access -vs- microsecond– Current disks >3ms
5Copyright 2009 MySQL AB The World’s Most Popular Open Source Database
A bigger buffer pool matters
6Copyright 2009 MySQL AB The World’s Most Popular Open Source Database
Scalability
• A huge draw back to the standard MySQL InnoDB storage engine is its difficulty to scale unless you plan for it up front... – Many clients find out too late they need to rewrite
portions of their applications to get better performance: • Read/Write splitting • Sharding
– Tools over the years have popped up trying to address these issues
• MySQL-Proxy• Spock-Proxy• HiveDB
7Copyright 2009 MySQL AB The World’s Most Popular Open Source Database
Real World Example
• Your database has grown up to 120GB, of which 50GB is “hot” active most of the time
• Your server is limited to 32GB, you are out of the InnoDB buffer pool so disk access is heavy
• Database performance suffers from the disk access• You have two other decommissioned servers with 16GB of
memory
8Copyright 2009 MySQL AB The World’s Most Popular Open Source Database
What can you do?
• Buy a server with 64 GB or 128 GB of ram?– Simple– Costly
• Shard the database to the other 2 servers?– Complex – Application rewrite for sharding needed
• Find a way to use the memory of the old servers?– How?
9Copyright 2009 MySQL AB The World’s Most Popular Open Source Database
Presenting Waffle Grid!
• What is Waffle Grid?– Waffle Grid is a distributed level 2 cache for Innodb
based on memcached
10Copyright 2009 MySQL AB The World’s Most Popular Open Source Database
What does that mean?
It means you can dramatically increase the size of the Innodb buffer pool by using the memory in other remote servers!
11Copyright 2009 MySQL AB The World’s Most Popular Open Source Database
How does it work?
• When the buffer pool is full, InnoDB needs to free the LRU page• The LRU page is valid, there is just no enough memory to hold it• Why just freeing it... send it to memcached• Before reading a page on disk, ask memcached first and if
lucky, save a disk IO operation
12Copyright 2009 MySQL AB The World’s Most Popular Open Source Database
How does it work
13Copyright 2009 MySQL AB The World’s Most Popular Open Source Database
How does it work (cont.)
14Copyright 2009 MySQL AB The World’s Most Popular Open Source Database
Optimal Configuration
• Network speed! 100Mb/s is too slow• Network speed! 1000Mb/s is ok• Network speed! Faster then 1000Mb/s is best• Waffle is going to work best if you give it a dedicated private
connections between servers specifically for memcached access
15Copyright 2009 MySQL AB The World’s Most Popular Open Source Database
Disclaimers
• I feel like the drug company putting out those disclaimers...• Waffle Grid will NOT! Directly boost write performance on your
disk, by reading less it can free up resources that may enable more writes...
• If your right constrained already, your mileage may very... • When testing out the dolphin interconnects we hit a write
constraint, the faster interconnects could not help because of it. • Waffle Grid is not going to help overcome CPU issues
bottlenecks
16Copyright 2009 MySQL AB The World’s Most Popular Open Source Database
Special Modified Memcached
• MRU -vs- LRU– Memcached also has a LRU logic– The problem is, memcached is a level 2 cache– Retrieving a block from memcached invalidates the block,
the good version is now in the buffer pool– A memcached GET must put the block at the top of the LRU
list (MRU logic)
– Patch available with WaffleGrid
17Copyright 2009 MySQL AB The World’s Most Popular Open Source Database
Benchmarks
• Tools– DBT2 – TPC like benchmark– Sysbench generic testing tool which includes OLTP tests
18Copyright 2009 MySQL AB The World’s Most Popular Open Source Database
Benchmarks
19Copyright 2009 MySQL AB The World’s Most Popular Open Source Database
Benchmarks
20Copyright 2009 MySQL AB The World’s Most Popular Open Source Database
Benchmarks – Interconnect speed
21Copyright 2009 MySQL AB The World’s Most Popular Open Source Database
Benchmarks – faster disk?
22Copyright 2009 MySQL AB The World’s Most Popular Open Source Database
Dolphin Interconnect Latency
23Copyright 2009 MySQL AB The World’s Most Popular Open Source Database
Dolphin Interconnect
500K trans 500K Trans 1M Trans0
200
400
600
800
1000
1200
1400
1600
503
14411498
502 1437 1497
504
14431502
464 898 783
463
899773
461 902 778
1gbe -vs- Dolphin Interconnects
Sysbench ReadOnly Test
Dolphin #1 Dolphin #2 Dolphin #3 1gbe #1 1gbe #2 1gbe #3
24Copyright 2009 MySQL AB The World’s Most Popular Open Source Database
EC2 Unique Challenges
• Network is shared on EC2• Boxes are shared• Boxes tend to be underpowered• Shared environment means slowdowns are a fact of life• Performance is similar to a 100Mb/s connection
25Copyright 2009 MySQL AB The World’s Most Popular Open Source Database
Waffle Grid on EC2
26Copyright 2009 MySQL AB The World’s Most Popular Open Source Database
Waffle Grid Everywhere!
• Waffle Gird can be deployed in many situations. – Those looking for the fastest possible performance will
benefit from the ability to heavy lift huge datasets into memory
– Those who are on a budget can reuse hardware and cobble together a Waffle Grid with many nodes boosting performance without incurring additional costs
– Waffle Grid can be deployed in the cloud, enabling cloud computing users to boost performance on demand.
27Copyright 2009 MySQL AB The World’s Most Popular Open Source Database
Status and roadmap
• Cache coherence between startup– Coded and partially tested, added a prefix to the memcached
key• Dynamic memcached server list
– Almost there– Allow to add more memory on the fly– Will invalidate current memcached cached pages
• Better error handling– Partly done
28Copyright 2009 MySQL AB The World’s Most Popular Open Source Database
Status and roadmap (2)
• Move from patches to a full branch– First port, the InnoDB Plugin– On launchpad: lp:~y-trudeau/wafflegrid/waffleGrid-Innodb-
plugin• Drizzle/Innodb Plugin port
– Should be straightforward from the InnoDB Plugin• Other engines?
– PBXT, Falcon• BETTER PERFORMANCE!
– EC2, High Latency environments, better hits rates
29Copyright 2009 MySQL AB The World’s Most Popular Open Source Database
Waffle Grid quick start
• Grab libmemcached-wafflegrid, compile and install – Branch lp:~yonkovim/wafflegrid/libmemcached
• Grab a recent MySQL source tree– http://dev.mysql.com/downloads/mysql/5.1.html#source
• Grab Waffle Grid for the InnoDB plugin– bzr branch lp:~y-trudeau/wafflegrid/waffleGrid-Innodb-plugin– Need bzr version 1.6 or higher
30Copyright 2009 MySQL AB The World’s Most Popular Open Source Database
Waffle Grid quick start (2)
• Replace the directory storage/innobase by the Waffle Grid code• Grab the compile script
– http://bazaar.launchpad.net/%7Ey-trudeau/wafflegrid/main/annotate/head%3A/BUILD-scripts/compile-pentium-max-memc
• Compile and install MySQL• Grab memcached version for Waffle Grid
– branch lp:~yonkovim/wafflegrid/memcached
31Copyright 2009 MySQL AB The World’s Most Popular Open Source Database
Waffle Grid quick start (3)
• Compile and install memcached• Start memcached
– From a non-root account: “memcached -d”• Start MySQL• From mysql client shell
– set global innodb_memcached_enable = 1;– show innodb status\G will keep give you stats about Waffle
Grid puts/hits/misses and timing
32Copyright 2009 MySQL AB The World’s Most Popular Open Source Database
Get Involved?
• http://www.wafflegrid.com/• https://code.launchpad.net/wafflegrid• Contributions are welcome
33Copyright 2009 MySQL AB The World’s Most Popular Open Source Database
Questions ?