Best practices around Dynacache

Jonathan MarshallWebSphere Technical Professionalmarshalj@uk.ibm.com

2

Acknowledgements

Based on presentations by:

Geoff Tindall – WebSphere level 2 support

Rama Boggarapu – WebSphere level 2 support

3

Agenda

Why do I need caching?

What is Dynamic Cache?

A peak under the covers of WebSphere Application Server

Replicating the Dynamic Cache

Monitoring the Cache

Troubleshooting Tips

Where do we go from here?

4

Why Cache?

(far away)

(near)

(happy)

A cache allows you to get stuff faster and helps youavoid doing something over and over again

(which may be redundant and may not make sense)

5

Why Cache?

Improving the performance

• Tuning the application?

• Tuning the environment?

• Tuning the backend resources?

• Not doing something at all?

6

Introducing Dynamic Caching Services (dynacache)

7

Enabling Dynamic Cache

8

What does it cache?

Each CacheEntry will have CacheId and CacheData

In addition to CacheId and CacheData, CacheEntry will have other information like dependencyId’s, timeout information, replication type and various other information.

CacheMonitor.ear application can be used to view most of the information about a CacheEntry.

9

Where does it cache data? The Cache Instance

An application uses a cache instance to store, retrieve, and share data objects within the dynamic cache.

Two types of custom cache instances can be configured:

Servlet Cache

• Cachespec.xml

Object Cache

• API based cache, com.ibm.websphere.cache.DistributedMap Three methods to create custom cache instance

Administrative console (under Resources)

Properties file (cacheinstances.properties)

Resource reference in web.xml

10

Dynamic Cache Configuration for Default Cache Instance

11

Implementation inherited from IBM Research

Disk cache size can be controlled in terms of space on the disk and # of entries.

LRU & Size based Disk eviction algorithms can configured to specify the criteria for eviction.

3 Performance modes HIGH, CUSTOM & LOW depending on the JVM free heap space available.

– High ..Buffers all disk metadata in memory.

– Custom/Balanced… Buffers some/most disk metadata in memory.

– Low … Buffers NO metadata in memory. Note this will be deprecated in v8.

FlushToDiskOnStop provides for the cache to be persisted.

One of the strongest features of Dynacache. Differentiates from other open source solutions as well as eXtreme Scale.

Disk Offload

12

Disk Offload

Configure a new cache instance

Resources – Cache Instances – Servlet Cache Instance – new

A jndi name must be given to the cache instance which will be used to refer back to the instance in cachespec.xml.

Check ‘Enable disk offload’ and specify a disk off load location, cache size and cache entry limits.

13

How does it choose what to cache?

Cachespec.xml is a deployable XML cache policy file that contains caching rules

Placed in WEB-INF directory

Specify the cache-instance name where the cache rules to be applied

What to be cached?

• command/servlet/webservice/JAXRPCClient/static/portlet

How to identify an item in cache?

• Cache-Id

Where to cache?

• Memory only/ Disk

When to invalidate?

• Timebased/invalidation rules

How to handle dependencies?

• Dependency rules for invalidation

14

Cache ID

A cacheID can be comprised of the following

• Request parameters and attributes

• The URI used to invoke the servlet or JSP

• Session information

• Cookies

• Pathinfo and servlet path

• Http header and Request method

• Servlet/JSP result caching can be used to cache both whole pages or fragments.

• Example

• /reference/Time.jsp:format=analog

• Web Service – getStockPrice but NOT setStockPrice

15

Example cachespec.xml (1/4) – Define the entry

<cache-entry> <name>/newscontroller</name>

<class>servlet</class> </cache-entry>

* http://publib.boulder.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=/com.ibm.websphere.nd.doc/info/ae/ae/welc6tech_dyn_dev.html

16

Example cachespec.xml (2/4) – Define the cache id

<cache-entry> <name>/newscontroller</name>

<class>servlet</class> <cache-id> <component id="action" type="parameter"> <value>view</value> <required>true</required> </component> <component id="category" type="parameter"> <required>true</required> </component> <component id="layout" type="session"> <required>false</required> </component> </cache-id></cache-entry>

* http://publib.boulder.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=/com.ibm.websphere.nd.doc/info/ae/ae/welc6tech_dyn_dev.html

17

Example cachespec.xml (3/4) – Define the dependencies

<cache-entry> <name>/newscontroller</name>

<class>servlet</class> <cache-id> <component id="action" type="parameter"> <value>view</value> <required>true</required> </component> <component id="category" type="parameter"> <required>true</required> </component> <component id="layout" type="session"> <required>false</required> </component> </cache-id> <dependency-id>category <component id="category" type="parameter"> <required>true</required> </component> </dependency-id></cache-entry>* http://publib.boulder.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=/com.ibm.websphere.nd.doc/info/ae/ae/welc6tech_dyn_dev.html

18

Example cachespec.xml (4/4) – Define the invalidation

<cache-entry> <name>/newscontroller</name>

... <invalidation>category <component id="action" type="parameter" ignore-value="true"> <value>update</value> <required>true</required> </component> <component id="category" type="parameter"> <required>true</required> </component> </invalidation>

</cache-entry>

* http://publib.boulder.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=/com.ibm.websphere.nd.doc/info/ae/ae/welc6tech_dyn_dev.html

19

Fragment Caching

• The content of A.jsp is composed of its own content plus 3 jsps(fragments) B.jsp,C.jsp and D.jsp

• Often a mix of static and dynamic

• consume-subfragmentstells the cache to store fragments

• <exclude> allows fragments to explicitly not be cached

Going under the covers:Data Replication Service

21

Data Replication Service - DRS

Data replication service (DRS) is an internal WebSphere Application Server component that replicates data.

Transport for sending data from one managed server to another.

Uses HAManager and DCS data stack frameworks to accomplish replication.

Has the notion of replicas

Data transfer channel can be encrypted. In practice no-one does this.

Benefits

Used by DynaCache, HTTPSession, Stateful Session Beans and SIP

Challenges

DRS Bootstrap will be expensive, if aggressive replication occurs during startup.

DCS uses a star topology for synchrony, can result in scalability bottlenecks.

22

High Availability Manager

•New feature in V6

•Collection of services

• Highly Available Singleton Services

• Low-level Replication Abstraction

• Shared State Bulletin Board

•Several runtime features depend on HA Manager services

•HA Manager runs in every single JVM in the cell

•HA Manager services are only available between JVMs that are part of the same core group

High Availability Manager

Distribution and Consistency Services (DCS)

Reliable Multicast Messaging (RMM)

Transaction Service Workload Management (WLM)

Data Replication Services (DRS)

MessagingEngine

On-DemandConfiguration (ODC)

23

HA Core Group Message Links

JVM

JVM

JVM

JVM

JVM

JVM

• N^2-N links•N=2, 2 links• N=3, 6 links• N=4, 12 links• N=5, 20 links• N=6, 30 links

With each additional JVM: • Geometrically greater messages flowing• More connections per JVM• More memory per JVM to hold messages

Optimizations to reduce message frequency and size are available in WAS 6.1

24

DRS – Important Tuning

• DRS shares the HA Manager with other services

• Under load, you may see large numbers of any of the following Distribution and Consistency Services (DCS) congestion messages in your SystemOut.log file:

DCSV1051W, a high severity congestion event for outgoing messages

DCSV1052W, a medium severity congestion event for outgoing messages

DCSV1054W, a medium severity congestion event for incoming messages

• Tuning directions are given in the InfoCenter: http://publib.boulder.ibm.com/infocenter/wasinfo/v6r1/index.jsp?topic=/com.ibm.websphere.nd.doc/info/ae/ae/trun_ha_cfg_replication.html

25

DRS – Important Tuning

Consider increasing the size of the default thread pool

– In larger configurations, the Default thread pool size should be increased.

– Doubling the thread pool size to 40 will likely be sufficient.

– However, when the number of application servers in a replication domain is greater 10 and the number of replication domain consumers in each application server exceeds 2, it may need to be increased more.

– This should keep the DCS traffic moving and avoid the timeouts.

The transport buffer size out of the box value may also be insufficient 

– For all the appservers doing replication,

– Click on Servers -> Application Servers -> <Your appserver> -> Coregroup service

– Change the transport buffer size to 100MB (or larger in more extreme loads). This changes the RMM buffer size.

We also recommend changing the IBM_CS_DATASTACK_MEG memory config parameter

– Servers --> Core Groups --> Core group settings -->then the settings for your core group (e.g. such as DefaultCoreGroup). Choose "Additional Properties" to specify a "Custom Properties". Key would be "IBM_CS_DATASTACK_MEG" with value in MB. Default value is 50.

Cache Replication

27

Replicating data across servers in a cluster

• Currently only dealt with the scenario where data is cached per JVM

• The Data Replication Service replicates data throughout the cell for various functions

• Dynacache

• HTTP Session

• Stateful Session Beans

• The scope of replication, the “Replication Domain” allows multiple deployment patterns

• Client/Server

• Client-only/Server-only

• Creating replication domains: http://publib.boulder.ibm.com/infocenter/wasinfo/v6r1/index.jsp?topic=/com.ibm.websphere.nd.doc/info/ae/ae/trun_drs_replication.html

28

Replication Topologies – Peer to peer

For dynacache and HTTP session replication

29

Replication Topologies – Client/Server

For HTTP session replication only

30

Replication Domain configuration

Single replica

– Limits replicated data to a single backup copy within the Replication Domain

– Reduces the total amount of redundant data held throughout the domain

– E.g. HTTP Session replication

Entire Domain

– Stores a backup copy within each DRS instance in the Replication Domain

– Maximises the redundancy of the data held throughout the domain

– E.g. Dynamic cache needs to use this

Specify

– Stores a specified number of backup copies within the Replication Domain

31

Configuring Replication for Default Cache Instance

32

Replication Types

In all of the following modes, the invalidations will be sent across servers for data consistency.

PUSH mode - cacheId and Cache Data will be sent to all servers in cluster

PULL mode (Not recommended) - Server requests the data from other servers in cluster, when not present in its cache

PUSH/PULL mode - Only cacheId is sent to all servers in cluster. When a server needs an entry in the Push/Pull table, it requests it from the server that has the copy.

Not Shared - Cache data will not be shared across servers. But invalidation events are sent to all servers.

Recommendation

Try PUSH and move to PUSH/PULL or even Not Shared if struggling to replicate

Monitoring the Cache

34

Monitoring the Cache – The CacheMonitor Application

Shipped with WebSphere application server install in the installableApps dir

35

Monitoring the Cache

Cache contents can also be displayed.

In our example, the cacheID is

/referenceWEB/Time.jsp:requestType=GET

36

Disk Offload

37

Extended Cache Monitor on developerWorks

http://www.ibm.com/developerworks/websphere/downloads/cache_monitor.html

Provides two functions that are not available with the cache monitor

Monitor and Manage contents of object cache

Monitor cache statistics for cache instances across all members of the cluster.

Enhancements made for the cache monitorin WebSphere Application Server V7:

look at the push-pull table associated with a cache instance

search memory contents, disk contents, and push-pull table using a regular expression

compare cache instances

38

Cache Statistics Collector and Visualizer

Statistics from DynaCache JMX MBean provide insight into the state, health, performance, composition & efficiency of the cache.

Collect cache statistics using DynaCacheStatisticsCSV.py wsadmin jython script.

Outputs statistics in CSV fileStatistics can be charted with Microsoft Excel or OpenOffice SpreadSheet.

http://www.alphaworks.ibm.com/tech/cacheviz/

Performance&

Troubleshooting

40

Dynamic Cache Common Sense Tips

WebSphere Application Server uses JVM memory to cache objects. Therefore, know how much memory can be allocated for the cache and set the cache size to the proper value.

Increase the priority of cache entries that are expensive to regenerate.

Modify timeout of entries so that they stay in memory as long as they are valid.

If the estimated total size of all cached objects is bigger than the available memory, you can enable the disk offload option.

Increase the cache size if memory allows.

41

Essential Dynacache Tuning Read the following document for cache replication tuning

• http://www-01.ibm.com/support/docview.wss?uid=swg27006431 Set the following Dynacache JVM Custom Properties

Name: com.ibm.ws.cache.CacheConfig.ignoreValueInInvalidationEvent Value: true

Name: com.ibm.ws.cache.CacheConfig.filterTimeOutInvalidation Value: true

Name com.ibm.ws.cache.CacheConfig.filterLRUInvalidation Value: true

Name: com.ibm.ws.cache.CacheConfig.cacheEntryWindow Value: 50

Name: com.ibm.ws.cache.CacheConfig.cacheInvalidateEntryWindow Value: 50

Name: com.ibm.ws.cache.CacheConfig.useServerClassLoader Value: true

Assume to use replication type of NOT_SHARED

Read the following Technote for Portal Server Cache Replication issues

• http://www-01.ibm.com/support/docview.wss?uid=swg21322640

Read the following Technote for Commerce server replication issues

• http://www-01.ibm.com/support/docview.wss?uid=swg21358672

42

Objects placed in cache are not replicated

Make sure the replication instance is launched (see SystemOut.log below)

Test with PUSH mode

If you are using PUSH_PULL mode, only CacheId will be pushed. Cache data will be pulled when needed

The following messages in SystemOut.log file tells Dynamic cache instance is initialized and replication service is launched. Notice a small delay, if application is looking for replicated data during the delay, it may not find it[2008-07-10 18:56:01:828 EST] 0000002c CacheServiceI I DYNA1001I: WebSphere Dynamic Cache instance named /cache/instance_one initialized successfully.[2008-07-10 18:56:02:828 EST] 00000042 DRSBootstrapM A CWWDR0001I: Replication instance launched : /cache/instance_one

See the following Technote for more information:

http://www-01.ibm.com/support/docview.wss?uid=swg21313480

43

Dynacache Congestion

Signs of Congestion:

(1) DCSV1051W/DCSV1052W: DCS Stack DefaultCoreGroup.MyCluster at Member CustomServer1: Raised a high severity congestion event for outgoing messages. Internal details are Total stored bytes: 67701476, Red mark is 41943040, Yellow mark is 37748736, Green mark is 8388608.

(2) HMGR0152W: CPU Starvation detected. Current thread scheduling delay is 109 seconds.

Congestion normally occurs when there is lot of data stored in cache and a member joins a HAManager view or if there are lot of invalidation events across servers.

Workarounds: Use the NOT_SHARED or PUSH_PULL replication mode

Filter out of unnecessary cache invalidation events by configuring custom properties:Name: com.ibm.ws.cache.CacheConfig.filterTimeOutInvalidation Value: trueName: com.ibm.ws.cache.CacheConfig.filterLRUInvalidation Value: true

44

Dynacache: Controlling DRS message size

When processing BatchUpates, the DRS message could be huge, which can lead to fragmentation issues and OutOfMemory errors.

The following custom properties introduced in PK32201 and PK35284 helps to control the size of DRS messages:com.ibm.ws.cache.CacheConfig.cachePercentageWindow – 2%com.ibm.ws.cache.CacheConfig.cacheEntryWindow – 50com.ibm.ws.CacheConfig.batchUpdateMilliseconds - 1 seccom.ibm.ws.cache.CacheConfig.cacheInvalidateEntryWindow – 50com.ibm.ws.cache.CacheConfig.cacheInvalidatePercentWindow -2%

Advanced topologies

46

Dynamic Cache Next Generation

47

WebSphere eXtreme Scale

WebSphere eXtreme Scale provides distributed object caching essential for elastic scalability and next-generation cloud environments. It processes massive volumes of transactions with extreme efficiency and linear scalability.

The WebSphere eXtreme Scale dynamic cache provider:

• It uses cheaper system memory instead of SAN or storage solutions

• Provides a scalable replicated cache with a configurable number of replicas. This eliminates the need to use broadcast data everywhere with DRS.

• Scales “elastically”. Additional WebSphere eXtreme Scale containers can be added at runtime. WebSphere eXtreme Scale automatically redistributes data partitions as new containers are added to the grid.

• Provides better caching qualities of service and control, than the default cache provider.

• Uses the same runtime monitoring and administration tools as the classic dynamic cache.

48

WCS JVM 1.5Gb heap

WCS App90% - 1.35Gb

WAS internals10% - 0.15Gb

BenefitsWith WXS, we offload the dynacache data store to WXS “grid”WCS estate needs 25% less memory for dynacache - potentially reduce WCS estatePerformance improvement from not needing disk I/ODisk not needed – cost savingsWCS throughput improvement through reduced “chatter” between JVMs and less GC overheadWXS can now provide much larger in-memory cache if desired by adding more JVMs (disk often constrained by size and contention limitsReplica WXS JVM gives the cache resilience

WCS JVM 1.5Gb heap

WCS App90% - 1.35Gb

WAS internals10% - 0.15Gb

WCS JVM 1.5Gb heap

WCS App90% - 1.35Gb

WAS internals10% - 0.15Gb

WCS JVM 1.5Gb heap

WCS App90% - 1.35Gb

WAS internals10% - 0.15Gb

WCS JVM 1.5Gb heap

WCS App90% - 1.35Gb

WAS internals10% - 0.15Gb

WCS JVM 1.5Gb heap

WCS App90% - 1.35Gb

WAS internals10% - 0.15Gb

WebSphere eXtreme Scale Example with Commerce

WXS Grid18Gb

Summary

50

Summary

• Dynamic Cache is a core service provided by WebSphere Application Server that can provide significant benefits to application developers for

– Web content – Servlets/JSPs, AJAX requests, Web Services

– DistributedMap API

– Java command objects

• Dynamic Cache provides a comprehensive monitoring support infrastructure

• Higher performance environments will need to do some tuning to ensure Dynamic Cache performs at its best

51

Reference

IBM Redbooks

WebSphere Application Server V6 Scalability and Performance Handbookhttp://www.redbooks.ibm.com/abstracts/SG246392.html

Mastering DynaCache in WebSphere Commerce

http://www.redbooks.ibm.com/abstracts/sg247393.html

WebSphere Dynamic Cache: Improving J2EE application performancehttp://www.research.ibm.com/journal/sj/432/bakalova.pdf

Backup

53

Cache Invalidation

It is essential that timely invalidations of cached content take place for the integrity of the website.

Mechanisms for invalidation are:

timeout or inactivity directives within cachespec.xml

group-based invalidation mechanism through dependency IDs.

Programmatic invalidation via the cache API 'com.ibm.websphere.cache.*’

The CacheMonitor

54

“Data Oriented”

“Application Oriented”

Session management

Elastic DynaCache

Web side cache

Event Processing

In-memory OLTP

Worldwide cache

Petabyte analytics

Data buffer

In-memory SOA

eXtreme Scale

DataPower XC10 Appliance

• Ultimate flexibility across a broad range of caching scenarios

• In-memory capabilities for application oriented scenarios

• Drop-in cache solution optimized and hardened for data oriented scenarios

• High density, low footprint improves datacenter efficiency

Elastic caching for linear scalabilityHigh availability data replication

Simplified management, monitoring and administration

55

Caching Rules in cachespec.xml Consider the simple JSP,

Display.jsp.

The request to Display.jsp returns itself, as the parent, plus the included fragment Time.jsp.

56

Caching Rules in cachespec.xml

consume-subfragments

• The consume-subfragments property tells the cache not to stop saving content when it includes a child servlet. The parent entry will include all the content from all fragments in its cache entry, resulting in one big cache entry.

• Use the <exclude> element to tell the cache to stop consuming for the excluded fragment and instead, create a placeholder for the include or forward. For example, exclude Time.jsp from the consume-subfragment, as follows…

57

Caching Rules in cachespec.xml<cache>

<cache-entry>

<class>servlet</class>

<name>/Display.jsp</name>

<property name="consume-subfragments">true

<exclude>/Time.jsp</exclude>

h

<cache-id>

<timeout>30</timeout>

</cache-id>

</cache-entry>

</cache>

58

Caching Rules in cachespec.xmldo-not-consume

As discussed, the consume-subfragments property tells the cache to save all content including child fragments.

A fragment can be excluded from the rules of the parent by using the do-not-consume property.

In this cachespec, Time.jsp is labeled do-not-consume and can specify its own timeout or other rules.

<cache> <cache-entry> <class>servlet</class> <name>/Display.jsp</name> <property name="consume-subfragments">

true</property> <cache-id> <timeout>30</timeout> </cache-id> </cache-entry> <cache-entry> <class>servlet</class> <name>/Time.jsp</name> <property name="do-not-consume">

true</property> <cache-id> <timeout>10</timeout> </cache-id> </cache-entry></cache>

59

Caching Rules in cachespec.xml

Inactivity.

While the timeout directive dictates how long content can remain in cache before being refreshed, the Inactivity directive can cause a refresh prior to the timeout when a page is not used frequently.

<cache-id>

<timeout>600</timeout>

<inactivity>30</inactivity>

</cache-id>

60

Dependency IDs

Dynamic Cache provides a group-based invalidation mechanism through dependency IDs.

A dependencyID identifies a cache entries dependency based on certain factors, such that when those factors occur they trigger an invalidation of all the cache entries that share this dependency.

An example of such a dependency could be the invalidation of a page which lists customer names. (Customer.jsp) Cached entries for this list should be invalidated when a customer is added to or removed from the list.

61

Troubleshooting Dynamic Cache

Dynamic Cache Trace

Use the WebSphere trace facility to review key trace points and verify expected caching behavior.

Enabling Trace Dynamic Cache issues can be traced using the following trace specification:

• Dynacache replication is disabled− *=info:com.ibm.ws.cache.*=all

• Dynacache replication is enabled− *=info:com.ibm.ws.cache.*=all:com.ibm.ws.drs.*=all

For information regarding trace setting please refer to the WebSphere information center or see this this link:http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21254706

62

Troubleshooting Dynamic CacheBinding to cache instance

ResourceMgrIm I WSVR0049I: Binding services/cache/diskoffload as services/cache/diskoffload

First request

CacheHook 3 handleServlet: absoluteUri = /referenceWEB/Display.jsp

EntryInfo 3 set id=/referenceWEB/Display.jsp:name=Fred:requestType=GET

FragmentCompo 3 setConsumeSubfragments: /Display.jsp consumeSubfragments=true

FragmentCompo 3 setDoNotConsume: /Display.jsp doNotConsume=false

CacheStatisti 3 CACHE: Cache Miss: /referenceWEB/Display.jsp:name=Fred:requestType=GET

CacheHook 3 CACHE MISS id: /referenceWEB/Display.jsp:name=Fred:requestType=GET

CacheHook 3 servicing /referenceWEB/Display.jsp:name=Fred:requestType=GET

FragmentCompo 3 saveCachedData uri=/Display.jsp

Next request:

CacheStatisti 3 CACHE: Local Cache Hit: /referenceWEB/Display.jsp:name=Fred:requestType=GET

CacheHook 3 CACHE HIT id:/referenceWEB/Display.jsp:name=Fred:requestType=GET

Invalidation due to cache timeout

Cache > internalInvalidateById() cacheName=baseCache id=/referenceWEB/Display.jsp:name=Fred:requestType=GET Entry

Cache < internalInvalidateById() id=/referenceWEB/Display.jsp:name=Fred:requestType=GET Exit

If a problem is to be submitted to IBM Support, please collect trace and configuration by following this Mustgather document http://www-1.ibm.com/support/docview.wss?uid=swg21193837