CACHING: tuning your golden hammer By Ian Simpson from Mashery: an Intel Company.
-
Upload
preston-richard -
Category
Documents
-
view
228 -
download
6
Transcript of CACHING: tuning your golden hammer By Ian Simpson from Mashery: an Intel Company.
What’s being covered (or not) Caching within the JVM Not focusing on external caches like
memcached Not focusing on sharding Both are important, but we only have an
hour
Motivation for this talk
Overall, caching is great! More performance! Greater scalability! When do we have too much of a good
thing? How do we know if/when we’re doing it
wrong?
When is more not better?
Ultimately caching isn’t free (but is usually cheap)
Some caches are not effective (low cache/hit ratio)
Some caches hold on to too much data Some caches cause resource contention If used incorrectly, caches can destroy
performance
Memory: abundant but not free When we cache, we store data in
memory Generally a replacement for fetching
data from (slow) persistent storage Servers have lots of memory! The JVM only handles so much memory
gracefully*
* This varies based on the garbage collection strategy and JVM vendor, which is beyond the scope of this talk
JVM heap
Split between young, tenured, and permanent
Objects start in young gen Most objects in young gen are short lived
and GC’d Objects in young that survive GCs go to
tenured Objects in tenured are long lived (e.g.
cached data)
Heap o’ trouble
Tenured grows as more objects survive GCs cause “stop the world” events, or
pauses GCs on tenured get more costly the
larger it gets Long GC pauses can cause performance
problems Full heap can cause thrashing (lots of
GCs)
Cache structure and data retention Usually caches are maps of keys to
values Under the covers, often Map<K,V> Anything you put in a Map is strongly
referenced Any reachable strong reference can’t be
GC’d Caches need to be reachable to be
useful Caches don’t automagically clean up
stale data
How do we solve heap issues? Don’t cache more than is necessary
Cache/hit ratio should be high (> 80%) Understand how and when data is used Estimate how large your data is or will
be Use background thread to clean stale
data Guava has support for this
Can I just not use the heap?
Yes! You can do off heap caching! No! It’s not particularly straight forward! On 32-bit JVM, 4GB (or 2) – max heap
available On 64-bit JVM, all available memory –
max heap Space is part of the Java process Not managed by the garbage collector
What’s so difficult about off heap? You have to manage your own memory Allocated as a block via
java.nio.ByteBuffer Can also be allocated via
sun.misc.Unsafe You need to know where things are in the
array Serialization into and deserialization
from byte[] Cost to allocate block of memory
Blocking vs non-blocking cache Cache miss means data has to be loaded Loading data means blocking threads
requesting it Generally unavoidable unless returning
nothing is OK Cache hit on stale data means reloading Blocking on reloading data can be
optional Return stale data, reload asynchronously
Blocking cache: pros and cons Pros
Stale data is predictably stale Less resource intensive than loading
asynchronously Cons
Can have huge latency on popular keys across threads
Read mutexes can result in serial read performance
Better with many keys and low key contention
Non-blocking cache: pros and cons Pros
Can return stale data and load in the background
No read mutex required! Cons
Requires a thread pool for asynchronous loading
Data is unpredictably stale Better with fewer keys and high key
contention
Which is which? What do I use? EhCache is a blocking cache Guava cache can be either Neither is the best choice in all cases Data characteristics will help you decide Not sure what to use? Talk to your
coworkers!
Example cases
Product catalog with millions of items Blocking
Subset of products on the homepage Non-blocking
Subset of products on sale for limited time Questionable – contention and staleness
both important Split data on TTL, static vs dynamic Refresh price/inventory frequently and
asynchronously
Local vs distributed caching
Local means each server has its own cache
Distributed means multiple servers share cache
Local is very simple Distributed is very complex Greater detail of distributed is outside
scope
Local: pros and cons
Pros Basically just a decorated Map No serialization overhead Good ones available for free
Cons Less efficient: more cache misses and
memory usage In large clusters, cache/hit can suffer
greatly In clusters, consistency becomes a problem
Distributed: pros and cons
Pros Fewer cache misses in clusters; improves with
size Consistency across cluster Great for splitting up large data sets
Cons Complex: more data management and network
chatter Serialization overhead for remote retrieval Enterprise solutions cost $$$, free ones more
barebones
Which one do I use?
Again, no one perfect solution; depends on data
Local is quick and dirty Distributed is better for large data sets Distributed is usually better for clusters Distributed comes at a cost of
complexity and $$$ Not sure which one? Talk to your
coworkers!
Example cases
CMS with large view templates Distributed (size and consistency)
Display data like greetings, canned messages, etc Local (small, changes infrequently)
Session store for user behavior Questionable – consistency and resource
concerns Not mission critical data, but good to have Local with sticky sessions an option
Questions you should ask
How expensive is it to load? Size: IO/memory bound; electrons only
move so fast Cost: high DB or CPU load means you must
cache Frequency: death by 1000 cuts
How often will I hit cache? Example: static content often means high
cache hit ratio Example: highly diverse data means low
cache hit ratio
Preparation through estimation Estimate how data will grow
Number of new records per month? How much is considered active?
Estimate how diverse data will become Thousands of unique entries? Millions? Subset of data that is more popular?
Work with product owners on expected use of data
A recap: fundamentals
Caching is great, but it’s not free Get to know your data Consider how much memory your app
will use Consider how expensive it is to load data Take cache/hit ratios seriously Not everything needs a cache
A recap: choices to make
Non-blocking: great for bounded data Blocking: great for diverse data Non-blocking: uses more resources,
more stale data Blocking: bad when keys have high
contention Local: simple, cheap, but can impact
memory more Distributed: great for big data sets &
large clusters
Getting runtime info
Track cache statistics via JMX Cache hits and misses Total objects in memory
Tools like Nagios and AppDynamics can track JMX
Guava doesn’t have JMX, but it’s easy to add
BUT WHICH ONE DO I USE?!
Ultimately, you’ll have to figure it out … with the help of your coworkers Before you can decide on what to use,
you need to understand the characteristics of your data
Shameless self promotion
Website: http://www.theotherian.com Has several cache and memory related posts
Github Gists: http://gist.github.com/theotherian Lots of code samples (referenced in blog posts)
Interested in these problems? Mashery is hiring! http://mashery.com/careers API management for companies big and small Scalability is core to our business