Finding Java memory leaks in WebSphere Application Server, Part 1: Using Hprof and IBM heapdumps

Summary This paper explains how to diagnose and find the source of JavaTM memory leaks on IBMR WebSphereR Application Servers 3.5, 4.01 and 5.0 in the distributed and z/OS environments. It presents a general methodology, explains how to gather diagnostic data, describes tools for analyzing the data and gives examples of use.

Table of contents • Summary • Introduction

• Who should read this paper • How to read this paper

• Java memory leaks • Methodology

• Identify the symptoms • Turn on verbose garbage collection • Turn on profiling • Analyze the trace

• Sun and IBM heapdump differences • Profiling the heap with Hprof

• Enabling Hprof output • WebSphere Application Server 3.5 and 4.01 simple configuration for z/OS • WebSphere Application Server 4.01 J2EE Server for z/OS • WebSphere Application Server 3.5 distributed (version 3.5.6 and beyond) • WebSphere Application Server 4.x distributed • WebSphere Application Server 5.0 distributed

• Default behavior • Performance considerations • Creating and reading heapdumps using Hprof

• Hprof advanced usage • Understanding the JVMPI • Hprof options

• Understanding Hprof output • Using the HAT tool to intepret heapdumps

• Profiling the heap with IBM heapdumps • Is it IBM_HEAPDUMP or IBM_HEAP_DUMP?

• Enabling IBM heapdumps on WebSphere Application Server 5.0 distributed • Enabling IBM heapdumps on WebSphere Application Server 4.x distributed

• Invoking IBM heapdumps • Invoking with a signal

• On distributed Unix, AIX and Linux

• On Windows • Invoking automatically on an OutOfMemoryError • Invoking through special code

• Reading IBM_heapdump output • Interpreting IBM_heapdump output using the HeapRoots tool

• Memory and performance problems with HeapRoots • Using HeapRoots in interactive mode • An example of using HeapRoots to diagnose a memory leak • HeapRoots Summary

• Analyzing IBM dumps interactively with HeapWizard • Using HeapWizard • An example of heap analysis • HeapWizard Summary

• Conclusion • About the authors

Introduction This paper is the first in a two-part series exploring how to diagnose Java memory leaks on various IBM platforms. Due to the wide diversity of tools and techniques available, the authors have been forced to make some drastic choices about which ones are presented in these papers. Tools and techniques that are not presented within the scope of the paper are by no means invalid.

Who should read this paper

The intended readers are the developers and troubleshooters of Java applications or J2EE application servers.

How to read this paper

The paper discusses the various diagnostic tools and the types of files on which they operate. It offers examples of tool usage and output. If a new version of the tool is available by the time you read this, you might find that the output or the usage is slightly different.

The authors have tried to make it easy to jump right to the part of the paper that might specifically interest a reader. This means that some information is repeated.

Sometimes, a problem has been identified that is still not resolved at publication time. In this case, the problem is mentioned in a note.

The paper discusses each platform separately when needed. For the purpose of this document, distributed means a platform other than z/OS.

Memory problem resolution is a technology still in its infancy. The approach taken and the tools available vary considerably depending on the Java Virtual Machine (JVM) origin (Sun or IBM), the operating system platform and the problem type (performance, constraint, or leak). This document walks you through many of the resources available today. Be sure to contact your IBM Support representative and check WebSphere newsgroups periodically for updates to existing tools and to learn about emerging problem resolution technology.

A great reference for debugging aids and techniques for IBM JDKs can be found at http://www.ibm.com/developerworks/java/jdk/diagnosis/.

Java memory leaks Most people think of memory problems in terms of memory leaks and their objective is to locate the leaking object. However, there are actually four different categories of memory problems with similar or overlapping symptoms, but different causes and solutions: performance, resource constraints, Java heap leaks and native memory leaks. Performance problems are usually associated with excessive object creation and deletion, long delays in garbage collection, excessive operating system page swapping, and more. Resource constraints are caused by either not enough memory or memory too fragmented to allocate a large object -- this can be native or Java heap, but is usually Java heap-related.

Java heap leaks are the classic Java memory leak, in which Java objects are continuously created without being released. This is usually caused by latent object references. Native memory leaks are associated with any continuously growing memory utilization that is outside the Java heap, such as allocations made by JNI code, drivers or even JVM allocations. This document will limit its scope to Java heap leaks.

Methodology

The methodology for finding Java memory leaks is straightforward:

• Identify the symptoms • Turn on verbose garbage collection • Turn on profiling • Analyze the trace

Identify the symptoms

While Java contains a built-in garbage collector to collect objects that are no longer used, there is still a possibility that a program can leak memory. Memory leak is a possible suspect when:

• The Java process, for example, the WebSphere Application Server (Application Server), unpredictably slows down, freezes, or stops, even when Administrative Server (3.5 and 4.0 only) and Web Server are fine.

• The size of memory used by the Java process, such as Application Server, steadily rises and is not kept steady by garbage collection.

Eventually the Java process throws a runtime exception, java.lang.OutOfMemory, which is a clear indicator that memory resources have been exhausted.

You need to distinguish between a normal memory exhaustion from a leak. If a Java application requests more storage than the runtime heap offers, it can be because of poor design. For instance, if an application creates multiple copies of an image or loads a file into an array, it will run out of storage when the image or file are very large. This is a normal resource exhaustion. The application is working as designed, even though the design is boneheaded.

But if an application steadily increases its memory utilization while processing the same kind of data, you might have a memory leak.

Turn on verbose garbage collection

One of the quickest ways to assert that a memory leak is occurring is to use verbose garbage collection. Memory constraint problems can usually be identified by examining patterns in the verbosegc output. Java heap leak resolution involves using the verbosegc output along with some JVM-specific techniques.

The java command provides an input argument, -verbosegc, which generates a trace each time the garbage collection (GC) process is started. That is, as memory is being garbage-collected, summary reports are printed to standard error. The output shows the JVM trying to allocate memory as needed.

Typical output looks as follows:

Each block (or stanza) in this GC trace file is numbered in increasing order. "Allocation Failure" itself is normal. To make sense of this trace, you should look at successive Allocation Failure stanzas and look for freed memory, bytes and percentage, decreasing over time while total memory (here, 19725304) is increasing. These are typical signs of memory depletion. Once this is verified, heapdumps, which we’ll describe later, should be generated to isolate the cause of the leak.

Memory fragmentation can also occur. When the heap is almost full and an allocation request cannot be satisfied, the GC will trigger. However, the freed memory fragments may not be contiguous. As a result, an Out of Memory exception will occur even if the heap says it contains enough free space. Figure 1 illustrates this case.

Figure 1: A situation where a fragmented heap cannot satisfy an allocation request

Turn on profiling

The different JVMs offer ways to generate trace files that reflect the heap activity. This is called profiling the heap. This is not to be confused with other types of profiling, such as performance measurements, which are outside the scope of this paper.

Heap profiling produces a trace file, generally quite large, that contains information about the type and size of objects allocated in the heap. Once this trace has been collected, you can analyze it to pinpoint the problem.

Analyze the trace

This paper focuses on two different heap trace formats, the Sun format known as Hprof and the IBM format known as IBM heapdump. The tools applicable to each format are different, but the idea is the same: find a block of objects in the heap that should not be there, and determine if these objects accumulate instead of being cleaned up. Of particular interest are transient objects that are known to be allocated each time a certain event is triggered in the Java application. The presence of many instances of objects that ought to exist only in small quantities generally indicates an application bug.

Finally, solving memory leaks requires the application programmers to review their code and fix the leak. Telling the coders what kind of objects are leaking is a great help and considerably speeds up the correction of the program. A common bug is to have a reference to a transient object involuntarily kept around in another object, preventing the garbage collector to delete this object and reclaim its space. This paper won’t venture into the vast territories of Java debugging.

Sun and IBM heapdump differences All 1.2 or later JDKs (except for iSeries) have a built-in standard feature, called "Hprof." This feature writes a map of memory usage, or "heapdump," to a file which can be manually inspected or fed into a software tool for analysis. Hprof dumps display objects, their addresses and sizes, and Java stacks, which show where in code the objects were allocated. This document explains how to use Hprof to create and interpret heapdumps in the following sections.

IBM JDKs, which power most versions of WebSphere Application Server (including all JDKs for WebSphere Application Server 5.01 and later, except for iSeries), include an additional utility for creating an IBM-specific format of heapdumps. The IBM heapdump is a superset of the Hprof format. You cannot feed an IBM heapdump to an Hprof tool such as Heap Analysis Tool (HAT). Conversely, you cannot feed an Hprof dump to an IBM heapdump tool. You must choose the format (Hprof or IBM) of the heapdump generated by your JVM.

IBM heapdumps, like Hprof dumps, list allocated objects and their addresses and sizes. Unlike Hprof dumps, IBM heapdumps do not show where in the Java code objects are allocated, or relate objects that were allocated at the same time or in the same Java method. Like Hprof dumps, they show object relationships, that is, who has pointers to whom. This can be important, since it is these relationships which prevent objects from being garbage-collected. The section Profiling the heap with IBM heapdumps explains how to generate and interpret IBM heapdumps.

In general, Hprof dumps are more direct in pinpointing memory leak issues, because they show where in the code objects are allocated, and show combined memory size and occurrences for all objects allocated in the same code location. However, enabling Hprof causes significant performance degradation, which may make it unacceptable in a production environment and may even skew test results, whereas IBM heapdumps

require little runtime overhead.

Profiling the heap with Hprof The easiest way to obtain Hprof output is to add the -Xrunhprof option to the Java command line argument. For example:

java -Xrunhprof -X... -D.... com.mycompany.mypackage.myclass

The section below explains how to add the -Xrunhprof option to the Application Server JVM for the different levels of the Application Server and the various platforms.

Enabling Hprof output

The following sections describe how to enable Hprof output on the various platforms.

WebSphere Application Server 3.5 and 4.01 simple configuration for z/OS

Locate the application server's was.conf file.

1. Add this parameter to was.conf:

appserver.java.extraparm=-Xrunhprof:file=/some_writable_directory/name_of_output_file

Note: You must use the file option on z/OS. The application server will not write the Hprof output to a default working directory. 2. Restart the application server.

Note: Currently, this option does not produce the expected output, whether you send a kill -s SIGQUIT to the PID of the address space, or shut down WebSphere Application Server normally. This is being debugged by WebSphere and Java support.

WebSphere Application Server 4.01 J2EE Server for z/OS

1. Start the System Management console and connect to the target system. 2. Start a new conversation. 3. Expand the tree until you reach the target J2EE server. 4. Right-click on the J2EE server and select Modify. 5. Scroll down on the right panel until you reach the environment variables section. 6. Within the environment variables, scroll down until you see a blank entry. 7. Double-click inside this entry to display a dialog box for creating a new environment variable. 8. Enter the following environment variable:

Name: JVM_EXTRA_OPTIONS, value: -Xrunhprof 9. Click OK, then Save. 10. Validate, commit, and complete the tasks, then activate the conversation.

Notes:

• This procedure will write the java.hprof.txt file to your /tmp directory, not your working directory. • If you have more than one option to pass to the JVM, use the WAS_JAVA_OPTIONS environment

variable documented in APAR PQ59164.

Note: Currently, this option does not produce the expected output when sending a kill -s SIGQUIT to the PID of the address space. This is being investigated through PMR 09839,999. This option works if a normal shutdown of the J2EE server takes place via the MVS STOP command.

WebSphere Application Server 3.5 distributed (version 3.5.6 and beyond)

Locate the application server property page for each application server using the Admin Console:

1. Start the Admin Console. 2. Expand WebSphere Administrative Domain by clicking on the +. 3. Expand the target host. 4. Select the target server (for example, MyServer) and stop it. 5. In the right pane of the General tab, update the Command line arguments field with -Xrunhprof. 6. Optionally, add specifications to the Hprof option, as follows:

-Xrunhprof:depth=n,file=filename.txt

7. where n = the depth of Java stack traces you wish to view. A small depth (such as 5) will make the heapdump smaller and easier to manage.

8. Click Apply. 9. Restart the application server. 10. Trigger a thread dump using the following commands:

On Windows: DrAdmin -port portnumber -dumpThreads On AIX: kill -3 JavaProcessPID

WebSphere Application Server 4.x distributed

1. Start the Admin Console. 2. Expand WebSphere Administrative Domain by clicking on the +. 3. Expand Nodes. 4. Expand the node that hosts the problem application server. 5. Expand Application Servers. 6. Select the problem application server 7. Click the JVM Settings tab. 8. Click the Advanced JVM Settings tab. 9. Check Run Hprofs arguments. 10. Enter specifications in the Hprof arguments field, as follows:

depth=n,file=filename.txt

where n = the depth of Java stack traces you wish to view. A small depth, such as 5, will make the heapdump smaller and easier to manage.

11. Click OK, then Apply.

Note: If the application server is a member of a server group, then the Hprof settings will be disabled. The steps in this case are the same, except that the settings must be made on the server group properties instead of the application server.

12. Restart the problem application server, and trigger a thread dump using the following commands:

On Windows: DrAdmin -port portnumber -dumpThreads On AIX: kill -3 JavaProcessPID

If the absolute path was not given as part of the heap file name, look for it in the install_root/bin directory.

WebSphere Application Server 5.0 distributed

1. Start the Admin Console and navigate to Servers => Application Servers => server_name -> [Configuration Tab] => Process Definition => Java Virtual Machine.

2. Check Run Hprof and specify format=a in the box below that for Hprof Arguments. 3. Save the configuration and restart the application server. 4. Run the application that causes the leak. 5. Using the wsadmin command, get a handle to the problem application server as follows:

wsadmin>set jvm [$AdminControl completeObjectName type=JVM,process=server_name,*]

6. Generate a thread dump (which also triggers Hprof, if enabled):

wsadmin>$AdminControl invoke $jvm dumpThreads

This file will be written to the default working directory (for example, c:\winnt\system32 on Windows), unless it is overridden in the AdminConsole => targetserver =>General tab->Working directory field.

Default behavior

A header is written to java.hprof.txt when the Java process starts. A complete memory profile output, or heapdump, is appended when the Java process exits.

Incremental heapdumps are added if a signal is sent to the running Java process; for example, kill -3 JavaProcessPID. Alternately, this signal can be delivered interactively if the running process has an associated console by typing Ctrl+\ (on Solaris) or Ctrl+Break (on Windows).

On WebSphere Application Server 3.5 or 4.x distributed, you can also use the DrAdmin command:

DrAdmin -port portnumber -dumpThreads

If interactive mode is not practical, as is the case in server or daemon processes, the kill -3 option is the best alternative. The -Xrunhprof:net=host:port option is more difficult to use, because it requires a client process running on the specified host and listening on the specified port prior to starting the Java process. You can get an example of such a client from http://www.javasoft.com.

There are other ways to deliver a signal interrupt to a running application server.

Note: Every time DrAdmin is called with the -dumpThreads option, a complete heapdump is appended to the output file.

Performance considerations

It is our experience that enabling heap profiling slows down a Java process significantly. At the time of this writing, no quantification had been made to evaluate the magnitude of this performance degradation. A good

practice is to reproduce a memory leak condition in a test environment. You should take the performance degradation caused by enabling heap profiling into consideration and measure it while in pre-production before you enable it on a production system.

WebSphere Application Server 3.5 running on Win2000 showed a 2.2X performance degradation in an application server, when the default -Xrunhprof option was added. The base measurement was performed on a 1 GHz / 1 GB RAM ThinkPad. 1000 URL getRequests issued against a servlet that performed EJB queries took 19 seconds. When Hprof was enabled, the response time went up to 43 seconds.

WebSphere Application Server 5.0 running on the same set-up as above showed a base response time of 15 seconds. When Hprof was enabled, the response time degraded to 46 seconds, roughly a 3X increase. WebSphere Application Server 4.x running on Windows showed a 1.5X degradation in response time.

WebSphere Application Server 3.5 running on z/OS showed a 1.75X performance degradation. A WebSphere Application Server 4.01 basic configuration running on z/OS showed a 1.91X performance degradation. WebSphere Application Server 4.01 J2EE server running on z/OS showed a 1.25X performance degradation.

Creating and reading heapdumps using Hprof

Let's examine the Hprof command options and see how to use its output to find memory leaks.

Begin by looking at the statistical report toward the bottom of the heapdump, with the section titled SITES BEGIN:

SITES BEGIN (ordered by live bytes) Wed Apr 02 07:13:47 2003 percent live alloc'ed stack class rank self accum bytes objs bytes objs trace name 1 99.76% 99.76% 125829168 12 125829168 12 2695 [C 2 0.03% 99.79% 32776 2 32776 2 1366 [C 3 0.01% 99.80% 16392 2 16392 2 1364 [B 4 0.01% 99.81% 16032 102 16240 112 1 [C ... SITES END

The self column shows what percentage of live bytes are due to that row, and accum shows the percentage of memory due to all rows up to and including that row (which are presented in descending order by amount of memory allocated). Therefore, in the table above, 99.76% of all live, non-collectable, bytes are due to 12 character arrays, which were allocated at stack trace #2695. 0.03% of live bytes were due to character arrays allocated at stack trace #1366.

Live versus alloc'ed distinguishes non-collectable, currently live objects from all objects ever allocated, whether currently live or previously garbage-collected, at a particular call site. Look for rows that have live values nearly equal to alloc'ed values, which is a sign that an allocation site is responsible for a leak. Problems with memory leaks often surface in the SITES section. As seen above, one or two sites are often responsible for the vast majority of the total allocated memory.

Once a suspicious site has been found, look at its number in the stack trace column. Then go to the TRACE section in the file and look for that trace number. It will list the methods responsible for the allocation.

The difficulty with this information is that low-level objects, such as character arrays, float to the top. The really helpful objects such as heads of leaking data structure are far fewer in number, and will be somewhere far down the list. Locating those is sometimes difficult. This is where third party vendor tools come in handy.

Hprof advanced usage

The following section are paraphrased from IBM Developer Kit and Runtime Environment, Java Technology Edition,Version 1.3.1, Diagnostics Guide, SC34-6200-00, written by the IBM Java Technology Center in Hursley and Bangalore.

Understanding the JVMPI

The Java Virtual Machine Profiling Interface (JVMPI) is a two-way interface that allows communication between the JVM and a profiler. JVMPI enables third parties to develop profiling tools based on this interface. The interface contains mechanisms that enable the profiling agent to notify the JVM about the kinds of information it wants to receive, as well as a means of receiving the relevant notifications. Several tools are based on this interface, such as Jprobe, OptimizeIt, TrueTime, and Quantify. These are all third-party commercial tools, so IBM cannot make any guarantees or recommendations regarding their use. The Hprof profiling agent is based on this interface.

Hprof options

The command java -Xrunhprof:help displays the options available for Hprof:

Option Name and Value Description Default

heap=dump | sites | all heap profiling All

cpu=samples | times | old CPU usage Off

monitor=y | n monitor contention N

format=a | b ascii or binary output A

file=filename write data to file java.hprof (for binary), java.hprof.txt (for ascii)

net=host:port send data over a socket write to file

depth=size stack trace depth 4

cutoff=value output cutoff point 0.0001

lineno=y | n line number in traces? Y

thread=y | n thread in traces? N

doe=y | n dump on exit? Y

Example:

java -Xrunhprof:cpu=samples,file=log.txt,depth=3 FooClass

Here is a detailed description of these options:

heap=dump|sites|all This option helps in the analysis of memory usage. It tells hprof to generate stack traces, from which you can see where memory was allocated. If you use the heap=dump option, you get a dump of all live objects in the heap. With heap=sites, you get a sorted list of sites with the most heavily allocated objects at the top.

cpu=samples|times|old The cpu option outputs information that is useful in determining where the CPU spends most of its time. If cpu is set to samples, the JVM pauses execution and identifies which method call is active. If the sampling rate is high enough, you get a good picture of where your program spends most of its time. If cpu is set to times, you receive precise measurements of how many times each method was called and how long each execution took. Although this is more accurate, it slows down the program.

The cpu=old option provides an output format that is backward-compatible with an older version of the tool.

monitor=y|n The monitor option can help you understand how synchronization affects the performance of your application. Monitors are used to implement thread synchronization, so getting information on monitors can tell you how much time different threads are spending when trying to access resources that are already locked. Hprof also gives you a snapshot of the monitors in use. This is very useful for detecting deadlocks.

format=a|b The default is for the output file to be in ASCII format. Set format to b if you want to specify a binary format, which is required for some utilities such as the Heap Analysis Tool.

file=filename The file option lets you change the name of the output file. The default name for an ASCII file is java.hprof.txt. The default name for a binary file is java.hprof.

net=host:port To send the output over the network rather than to a local file, use the net option.

depth=size The depth option indicates the number of method frames to display in a stack trace (the default is 4).

thread=y|n If you set the thread option to y, the thread ID is printed beside each trace. This option is useful if it is not clear which thread is associated with which trace. This can be an issue in a multi-threaded application.

doe=y|n The default behavior is to write profile information to the output file when the application exits. To collect the profiling data during execution, set doe (dump on exit) to n and use one of the methods (wsadmin, DrAdmin or kill -3) described in section Enabling Hprof output.

Understanding Hprof output

The top of the output file contains general header information such as an explanation of the options, copyright, and disclaimers. A summary of each thread appears next.

You can see the output after using Hprof with a simple program, as shown below. This test program creates and runs two threads for a short time. From the output, you can see that the two threads called respectively apples and oranges were created after the system-generated main thread. Both threads end before the main thread. The address, identifier, name, and thread group name are displayed for each thread. You can see the order in which threads start and finish.

THREAD START (obj=11199050, id = 1, name="Signal dispatcher", group="system") THREAD START (obj=111a2120, id = 2, name="Reference Handler", group="system")

THREAD START (obj=111ad910, id = 3, name="Finalizer", group="system") THREAD START (obj=8b87a0, id = 4, name="main", group="main") THREAD END (id = 4) THREAD START (obj=11262d18, id = 5, name="Thread-0", group="main") THREAD START (obj=112e9250, id = 6, name="apples", group="main") THREAD START (obj=112e9998, id = 7, name="oranges", group="main") THREAD END (id = 6) THREAD END (id = 7) THREAD END (id = 5)

The trace output section contains regular stack trace information. The depth of each trace can be set, and each trace has a unique ID:

TRACE 5: java/util/Locale.toLowerCase(Locale.java:1188) java/util/Locale.convertOldISOCodes(Locale.java:1226) java/util/Locale.(Locale.java:273) Java/util/Locale.(Locale.java:200)

A trace contains a number of frames, and each frame contains the class name, method name, filename, and line number. In the example above, you can see that line number 1188 of Local.java (which is in the toLowerCase method) has been called from the convertOldISOCodes() function in the same class. These traces are useful in following the execution path of your program. If you set the monitor option, a monitor dump gives output that looks like this:

MONITOR DUMP BEGIN THREAD 8, trace 1, status: R THREAD 4, trace 5, status: CW THREAD 2, trace 6, status: CW THREAD 1, trace 1, status: R MONITOR java/lang/ref/Reference$Lock(811bd50) unowned waiting to be notified: thread 2 MONITOR java/lang/ref/ReferenceQueue$Lock(8134710) unowned waiting to be notified: thread 4 RAW MONITOR "_hprof_dump_lock"(0x806d7d0) owner: thread 8, entry count: 1 RAW MONITOR "Monitor Cache lock"(0x8058c50) owner: thread 8, entry count: 1 RAW MONITOR "Monitor Registry lock"(0x8058d10) owner: thread 8, entry count: 1 RAW MONITOR "Thread queue lock"(0x8058bc8) owner: thread 8, entry count: 1 MONITOR DUMP END MONITOR TIME BEGIN (total = 0 ms) Thu Aug 29 16:41:59 2002 MONITOR TIME END

The first part of the monitor dump contains a list of threads, including the trace entry that identifies the code the thread executed. There is also a thread status for each thread where:

• R = Runnable • S = Suspended • CW = Condition Wait • MW = Monitor Wait

Next is a list of monitors along with their owners and an indication of whether there are any threads waiting on them.

The Heapdump is the next section of the output file. This is a list of different areas of memory and shows how they are allocated:

CLS 1123edb0 (name=java/lang/StringBuffer, trace=1318) super 111504e8 constant[25] 8abd48 constant[32] 1123edb0

constant[33] 111504e8 constant[34] 8aad38 constant[115] 1118cdc8 CLS 111ecff8 (name=java/util/Locale, trace=1130) super 111504e8 constant[2] 1117a5b0 constant[17] 1124d600 constant[24] 111fc338 constant[26] 8abd48 constant[30] 111fc2d0 constant[34] 111fc3a0 constant[59] 111ecff8 constant[74] 111504e8 constant[102] 1124d668 ... CLS 111504e8 (name=java/lang/Object, trace=1) constant[18] 111504e8

CLS tells you that memory is being allocated for a class. The hexadecimal number following it is the actual address where that memory is allocated.

Next is the class name, followed by a trace reference. Use this to cross-reference the trace output and see when this is called. If you refer back to the particular trace, you can get the actual line number of code that led to the creation of this object.

The addresses of the constants in this class are also displayed and, in the above example, the address of the class definition for the superclass. Both classes are children of the same superclass (with address 11504e8). Looking further, you can see the class definition and name, which in this case turns out to be the Object class (a class that every class inherits from). The JVM loads the entire superclass hierarchy before it can use a subclass. Thus, class definitions for all superclasses are always present. There are also entries for Objects (OBJ) and Arrays (ARR):

OBJ 111a9e78 (sz=60, trace=1, class=java/lang/Thread@8b0c38) name 111afbf8 group 111af978 contextClassLoader 1128fa50 inheritedAccessControlContext 111aa2f0 threadLocals 111bea08 inheritableThreadLocals 111bea08 ARR 8bb978 (sz=4, trace=2, nelems=0, elem type=java/io/ObjectStreamField@8bac80)

If you set the heap option to sites or all (dump and sites), you also get a list of each area of storage allocated by your code. The sites that allocate the most memory are at the top:

SITES BEGIN (ordered by live bytes) Thu Aug 29 16:30:31 2002 percent live alloc'ed stack class rank self accum bytes objs bytes objs trace name 1 18.18% 18.18% 32776 2 32776 2 1332 [C 2 9.09% 27.27% 16392 2 16392 2 1330 [B 3 8.80% 36.08% 15864 92 15912 94 1 [C 4 4.48% 40.55% 8068 1 8068 1 31 [S 5 4.04% 44.59% 7288 4 7288 4 1130 [C 6 3.12% 47.71% 5616 36 5616 36 1 7 2.51% 50.22% 4524 29 4524 29 1 java/lang/Class 8 2.05% 52.27% 3692 1 3692 1 806 [L; 9 2.01% 54.28% 3624 90 3832 94 77 [C 10 1.40% 55.68% 2532 1 2532 1 32 [I 11 1.37% 57.05% 2468 3 2468 3 1323 [C 12 1.31% 58.36% 2356 1 2356 1 1324 [C 13 1.14% 59.50% 2052 1 2052 1 95 [B 14 1.02% 60.52% 1840 92 1880 94 1 java/lang/String 15 1.00% 61.52% 1800 90 1880 94 77 java/lang/String 16 0.64% 62.15% 1152 10 1152 10 1390 [C 17 0.57% 62.72% 1028 1 1028 1 30 [B 18 0.52% 63.24% 936 6 936 6 4

19 0.45% 63.70% 820 41 820 41 79 java/util/Hashtable$Entry

The following table identifies the class name that appears in the rightmost column for each type that can be allocated:

Type signature Data type

[Z boolean

[B byte

[C Char

[S Short

[I Int

[J Long

[F Float

[D Double

[L object array

In this example, Trace 1332 allocated 18.18% of the total allocated memory. This works out to be 32776 bytes.

The cpu option gives profiling information on the CPU. If cpu is set to samples, you get output containing the results of periodic samples during execution of the code. At each sample, the code path being executed is recorded and you get a report such as this:

CPU SAMPLES BEGIN (total = 714) Fri Aug 30 15:37:16 2002 rank self accum count trace method 1 76.28% 76.28% 501 77 MyThread2.bigMethod 2 6.92% 83.20% 47 75 MyThread2.smallMethod ... CPU SAMPLES END

You can see that the bigMethod() was responsible for 76.28% of the CPU execution time and was being executed 501 times out of the 714 samples. If you use the trace IDs, you can see the exact route that led to this method being called.

Using the HAT tool to interpret heapdumps

HAT is an interactive tool from Sun that helps you interpret heapdumps. It is available at developer.java.sun.com/developer/onlineTraining/Programming/JDCBook/perf3.html#profile. HAT analyzes binary heapdumps produced by Hprof running with the format=b option.

The binary Hprof output file can be passed as an input argument along with a free TCP/IP port number to the HAT program. HAT runs on any Java 2 JDK to display details of the heapdump in a browser. The browser should be aimed at http://localhost:port, where port is the above chosen TCP/IP free port. HAT provides for interactive following of objects to their source and finding instances of specific classes. Additionally, HAT allows viewing and following all Java root (static) objects and objects held within.

Profiling the heap with IBM heapdumps With the IBM JRE, you can enable high performance heap profiling and obtain heapdumps. This involves setting the process environment variable IBM_HEAPDUMP to true. The performance degradation is minimal, generally no more than a 1-5% decrease in response time.

IBM_HEAPDUMP is supported on all IBM JDKs (1.1.8 and up) on distributed platforms, which means that it is supported by the JDKs supplied with WebSphere Application Server 3.0.2.4 and above on the Windows, AIX and Linux platforms, as well as all WebSphere Application Server 4.0x and 5.0x versions.

Is it IBM_HEAPDUMP or IBM_HEAP_DUMP?

Either. To activate the Heapdump feature, you must set either the IBM_HEAPDUMP or the IBM_HEAP_DUMP environment variables to true before you start the JVM.

For releases of the SDK 1.3.1 prior to SR3, you are required to set the variables to true. For more recent releases, any value will work, as long as the variable is set. If you are in doubt, just set IBM_HEAPDUMP to true and it will work in every case.

When a heapdump is written to the heapdump file, you will typically see the following in the standard error (stderr) stream:

Writing Heap dump .... Written Heap dump to somepath.YYYYMMDD.HHMMSS.PID.txt

Enabling IBM heapdumps on WebSphere Application Server 5.0 distributed

To enable IBM heapdump output on Application Server 5.0 distributed, do the following:

1. Start the Admin Console. 2. Navigate to Servers => Application Servers => server_name -> [Configuration Tab] -> Process

Definition => Environment Entries. 3. Select New. 4. In the Name field, enter IBM_HEAPDUMP. In the Value field, enter true. 5. Save the configuration. 6. Restart the application server.

To obtain an incremental IBM heapdump, use the wsadmin command as follows:

1. From the bin directory, type wsadmin 2. Under wsadmin, type:

set jvm [$AdminControl completeObjectName type=JVM,process=server1,*] $AdminControl invoke $jvm dumpThreads

where server1 is the name of the application server. The dump file will be created in the WebSphere Application Server install directory under the name javacore.yyyymmdd.mmmmmm.nnnn.txt.

Enabling IBM heapdumps on WebSphere Application Server 4.x distributed

To enable IBM heapdump output on Application Server 4.x distributed, do the following:

1. Start the Admin Console. 2. Expand the WebSphere Administrative Domain by clicking on the +. 3. Expand Nodes. 4. Expand the node that hosts the problem application server. 5. Expand Application Servers. 6. Select the problem application server. 7. Click the General tab and select Environment. 8. Click Add. 9. In the Name field, enter IBM_HEAPDUMP. In the Value field, enter true. 10. Click Apply and restart the application server.

To obtain an incremental IBM heapdump, use the DrAdmin command as follows:

1. Look in the application server log file for DrAdmin port. For example:

[02.02.19 15:17:07:073 CST] 59edf8e DrAdminServer I WSVR0053I: DrAdmin available on port 1793

2. Wait for the problem to occur, if possible 3. On the command line, type:

DrAdmin -serverPort 1793 -dumpThreads

4. In WebSphere_root\bin or c:\winnt\System32, look for a javacorexxx.yyyyyyyyy.txt file with a timestamp matching the date and time when DrAdmin was executed.

Invoking IBM heapdumps

This section describes other ways of invoking IBM heapdumps.

Invoking with a signal

Another way you can invoke the IBM heapdump mechanism is by sending a signal to the JVM. This is similar to the way you can trigger a Java Hprof dump, as described in Profiling the heap with Hprof.

On distributed UNIX, AIX and Linux

On a Unix, AIX, or Linux system, send a SIGQUIT to the JVM by running the command:

kill -QUIT JVM_Process_ID

where JVM_Process_ID is the process ID of the JVM. The JVM will temporarily stop processing and invoke the heapdump mechanism.

On Windows

On Windows, you can issue a SIGINT by pressing CTRL+Break. However, most Java processes are background processes, especially those for WebSphere Application Servers; therefore, you should use the WebSphere Application Server tools in order to invoke the heapdumps:

• WebSphere Application Server v4: use DrAdmin • WebSphere Application Server v5: use wsadmin

Invoking automatically on an OutOfMemoryError

You can arrange for an IBM heapdump to be generated automatically when an OutOfMemoryError condition is encountered in the JVM by setting the IBM_HEAPDUMP_OUTOFMEMORY environment variable to true before launching the JVM.

If you have enabled verbosegc in the JVM, you should also see the message "totally out of heap space" right before the OutOfMemoryError appears and the IBM Heapdump is triggered. Here is an example of what you will see:

Writing Heap dump .... Written Heap dump to D:\Code\memtest\heapdump.20021104.163757.2312.txt java.lang.OutOfMemoryError

This feature works independently of the IBM_HEAPDUMP or IBM_HEAP_DUMP variables. It is available on the following JVM versions:

• Java 1.3: JDK 1.3.1 SR3 and above • Java 1.4: JDK 1.4.1 and above

Invoking through special code

You can invoke the heapdump mechanism with Java code. This is easily done by invoking the static method HeapDump() in com.ibm.jvm.Dump. In the following example, the code first calls System.gc() to invoke the garbage collector to clean the unreferenced objects off the heap and then calls HeapDump() to generate a dump:

System.gc(); com.ibm.jvm.Dump.HeapDump();

The code invokes the garbage collector before the heapdump because on older JVMs, an IBM heapdump also dumps objects that are ready to be garbage-collected. This pollutes the dump with useless objects. Starting with JVM 1.4.1, the garbage collection is automatically invoked right before the heapdump, which means that the heapdump never contains garbage-collectable objects.

This feature is available on the following JVM versions:

• Java 1.3: JDK 1.3.1 SR3 and above • Java 1.4: JDK 1.4.1 and above

Reading IBM heapdump output

Unlike an Hprof-type heapdump, an IBM heapdump file is simple and homogeneous: except for one

statement at the beginning of the file and one at the end, the entire content consists of one- or two-line entries, all in the same format.

The first line of the heapdump identifies the dumping JVM, as in this example:

// Version: J2RE 1.3.1 IBM Windows 32 build cn131-20021107

The last line displays totals of each category of allocation in the dump: classes, objects, arrays of primitives, and arrays of objects:

// EOF: Total: 212945 Classes: 5141 Objects: 142218 ObjectArrays: 16193 PrimitiveArrays: 49393 //

The body of the dump consists of entries in the following form:

hex-address1 [size] type object-name hex-address2 hex-address3 hex-address4 ... hex-addressN

where:

• hex-address1 is the location within the JVM of the allocated object. • size is the amount of memory it represents, in bytes. This is only the amount of memory allocated to

represent the object itself, and does not include memory used by other objects to which it refers. • type is the meta-class of the object: class, object, array of object, or array of primitives • object-name is the name of the class, or the name of the object, that was allocated • The next hex addresses (if any), indented on the next line, represent the locations of objects pointed to

by this object.

For example:

0x007ede18 [256] class java/lang/Package 0x00a24190 0x00a24200 0x00a242c8

means that the class java/lang/Package was loaded at location 0x007ede18, that it takes up 256 bytes, and that this class object holds references to three other objects. You can search the heapdump for details on these other objects. Searching on the address 0x00a24190, you'll find the following entry:

0x00a24190 [56] java/util/HashMap 0x00a24158

In other words, the class java/lang/Package holds a reference to an object of class java/util/HashMap, which takes up 56 bytes. This object in turn has a reference to another object at location 0x00a24158, which takes up an additional amount of memory.

Would removing the class java/lang/Package, through garbage collection, have removed this particular instance of java/util/HashMap from memory? Not necessarily. In order to determine this, you would have to search the dump for all objects that contained references to the HashMap's location of 0x00a24190.

Examples of the other kinds of objects represented in IBM heapdumps are arrays of primitives:

0x00a24570 [128] primitive array

and arrays of objects:

0x00a24158 [56] array of java/util/HashMap$Entry

Here are a few notes about the IBM heapdump files:

• There is no particular ordering in the entries, such as time allocated, size, or object relationship. • There is no combining of data by object type. Each allocation of a String object, for example, is

shown separately. • There is no indication of where, in the Java code, an allocation occurred. • IBM heapdumps do not contain GCRoot information. This means that:

• Objects ready for garbage collection will be included in the dump but not identified as such. • Objects referenced only by local methods and with JNI global references will show up as

independent trees (generally thousands more than really exist).

Summary: Unlike Sun heapdumps, IBM heapdumps do not map memory allocation to Java code, nor do they include tables that clearly show which threads are responsible for taking up the biggest chunk of memory. You'd have to be a cyborg, or seriously disturbed individual, to be able to read the raw file and understand what objects were related to a memory problem. However, on the plus side, IBM heapdumps represent a much lighter performance burden on the JVM and may therefore be more acceptable for diagnosing memory problems in a production environment.

But, you may ask, if you can't interpret them, what good are they? You need a tool to interpret them. That tool is HeapRoots, discussed in the following section.

Interpreting IBM heapdump output using the HeapRoots tool

The HeapRoots tool is a post-processor program that reads a dump produced with the IBM_HEAPDUMP option, collates information based on object size, occurrences, and links, and enables you to view the information in various ways. HeapRoots reads only heaps created using the IBM_HEAPDUMP option on the JVM; it does not process Hprof-type heapdumps.

To obtain the HeapRoots tool, go to http://www.alphaworks.ibm.com/tech/heaproots. Keep in mind that HeapRoots is provided strictly as-is.

Once you have the HeapRoots JAR file, you can invoked it at any command prompt as follows (a Java version 1.2 or later must be in the path):

java -jar HRnn.jar heapdumpfilename

where HRnn is the name of the HeapRoots jar file, which depends on the version. The output is large enough that you will want to redirect the output to a file or (on Unix) pipe it to a more or similar utility.

A fragment of HeapRoots output follows:

0x007ec118 [1,364,912/29,455] class java/util/jar/JarFile 2 children smaller than 1,048,576 total size/desc: 97,888/1,974 0x00951650 [1,266,856/27,480] java/util/Vector 0x02f066c0 [1,266,824/27,479] array of java/lang/Object 367 children smaller than 1,048,576 total size/desc: 1,265,288/27,112 0x0094fc60 [28,612,016/507,751] array of com/ibm/ws/classloader/ReloadableClassLoader$CacheEntry 6 children smaller than 1,048,576 total size/desc: 336/6 0x01fd6fe8 [28,611,968/507,750] com/ibm/ws/classloader/ReloadableClassLoader$CacheEntry 2 children smaller than 1,048,576 total size/desc: 184/1 0x034a9b10 [28,611,936/507,749] com/ibm/ws/classloader/JarClassLoader 10 children smaller than 1,048,576 total size/desc: 4,184/61 0x034a9a10 [28,607,696/507,679] com/ibm/ws/classloader/CompoundClassLoader 15 children smaller than 1,048,576 total size/desc: 263,632/3,087

This fragment shows two root objects, objects which have no parents, or objects which refer to them. These are the class jar file, and an array containing instances of the com/ibm/ws/classloader/ReloadableClassLoader$CacheEntry class. The JarFile class is a parent (including all of its descendants) of 29,455 objects, taking over 1 MB of memory.

The array of CacheEntry objects is responsible for 597,751 objects and more than 28 MB of memory. It points to seven objects directly, but of these, six take up less than the 1 MB threshold. In fact the collective size of these six objects, including their descendants, is only 336 bytes. The seventh object is an instance of com/ibm/ws/classloader/ReloadableClassLoader$CacheEntry, no doubt an element in the array. Obviously it is responsible for the bulk of memory held onto by its parent. It has three immediate children, of which one is, in turn, responsible for the rest of its parent's subtree. The numbers on the left help you keep track of how far into a root object's sub-tree you are.

By default, root objects are displayed in order by address rather than size.

If you were to browse the entire contents of the HeapRoots output, you would have to go quite far (at least in this case) to find what most of that 28 MB is being used for -- even deeper than the 64 levels to which HeapRoots by default limits its display! The HeapRoots command gives you options to refine what you want to see and how you want to see it. You can see these options by entering the command without a file name:

HeapRoots version 2.0.0 Usage: java -jar [opts] [opts] opts are: -e - file encoding (use ISO8859_1 for ASCII on OS/390) -t - set threshold of object size, default 1048576 -d - set max depth for output, root-depth=0, default 64 -a - only dump object at specified address -i - interactive use -v - verbose mode

You could iteratively display the output and re-enter the HeapRoots command with these options to get to the heart of a memory problem. However, a better, faster approach is to use the interactive mode of HeapRoots, described below, to refine the data.

Memory and performance problems with HeapRoots

HeapRoots itself uses a large amount of memory in order to construct its model of memory usage. The precise amount of memory it uses depends upon the data in the input heapdump. It makes sense to use HeapRoots on a machine with a large amount of main memory. If you still experience very slow performance, or HeapRoots exits with java.lang.OutOfMemoryError exceptions, try adding the -XmxN option to the invocation of HeapRoots, where N is the amount of memory, in megabytes, to give to the JVM running HeapRoots. A rule of thumb given by the HeapRoots author is to specify about 80% of the main memory amount of the machine on which you are running HeapRoots. For example, on a machine with 512 MB of main memory:

java -Xmx400m -jar Heaproot200.jar heapdump.txt

For more details on HeapRoots memory usage and troubleshooting in general, see the HeapRoots reference listed at the end of this topic.

Using HeapRoots in interactive mode

The interactive mode of HeapRoots enables the tool to build a tree of heap objects in memory, and enables you to query that tree quickly in various ways without reloading the raw heap file. It does not mean you can interactively explore a running JVM's heap; you need to specify an existing heapdump file you want to process.

To launch the HeapRoots command prompt and load a heapdump file, enter the HeapRoots command with the -i option:

java -jar heaproots.jar OutputFile.txt -i

where heaproot.jar is the name of the HeapRoots jar file on your system and OutputFile.txt is the file that will contain the results.

You will be rewarded with a set of statistics about your heap:

Comments : // Version: J2RE 1.3.1 IBM Windows 32 build cn131-20021107 // EOF: Total: 639922 Classes: 7908 Objects: 453837 ObjectArrays: 62321 PrimitiveArrays: 115856 // # Objects : 639,922 # Refs : 988,571 # Unresolved refs : 33 Heap usage : 36,790,984 Est. Heap size : 397,090,048 Extra stats : unavailable before processing Memory Usage : 23/30 mb

Est. Heap size is the total difference between the lowest and highest address in the dumped JVM's memory. Since an Application Server JVM typically uses multiple heaps in separate locations, this figure is liable to be several times the real amount of Application Server memory usage and should be ignored. Memory Usage denotes the memory used by the HeapRoots tool itself.

The first command you should execute after launching HeapRoots is the p command. This command has two effects:

• It causes HeapRoots to construct an object tree in memory for further analysis. • It creates a compressed state file containing the tree information, so that the next time you launch

HeapRoots against this dump, the tree will not have to be reconstructed. (HeapRoots will first look for the state file when it is launched).

Now you can filter your view of the dump in various ways. To see a basic picture of object trees for each root object, similar to what you would see when dumping the output non-interactively, execute the d command. To see a list of available commands, type help:

> help Help: oa/os/on/ot/od show info on objects by addr/size/name/total size/descendants ts/tc show info on types by total size/count gs/gc show gaps by size/count i show info on a single object p process d dump from roots/single object stats show stats save save state for quick reload clear clear processed data return repeat last command x exit

Enter: o[a,s,n,t,d], t[s,c], g[s,c], i, p, d, x or help for more info

Where do you start when trying to interpret a heapdump? A good first step is to view root objects, and their total tree sizes only. To do this, use the ot command.

> ot Enter name to filter on or '-' for no filterting [-] Enter combination of types to show, R for Roots, A for Artificial Roots, N for Non-Roots [R,N,A] RA Enter address range in format 'M','L-U','-U' or 'L-' [0x00000000-0xfffffff8] Enter range of lines to print in format 'M','L-U','-U' or 'L-' [1-25] Addr. Size Root-Owner Subtree Size Descend. Name --------------------------------------------------------------------------------- R 0x0094fc60 48 - 28,612,016 507,751 array of com/ibm/ws/classloader/ReloadableClassLoader$CacheEntry R 0x007ec118 256 - 1,364,912 29,455 class java/util/jar/JarFile R 0x007e5e18 256 - 806,928 15,020 class com/ibm/jvm/ExtendedSystem R 0x007e5d18 256 - 209,512 2,595 class java/lang/System R 0x13d00b18 256 - 105,824 120 class com/ibm/rmi/iiop/CDROutputStream R 0x01526308 32 - 66,128 478 java/util/HashMap$Entry R 0x05228760 32 - 65,744 466 java/util/HashMap$Entry R 0x02ee35b0 32 - 65,360 440 java/util/HashMap$Entry R 0x007e9018 256 - 53,504 1,558 class sun/io/CharacterEncoding A 0x04a47228 72 - 51,312 56 com/ibm/ws/management/connector/soap/SOAPConnection R 0x029d91b8 24 - 51,288 1,791 array of java/util/ResourceBundle R 0x0131d628 64 - 49,600 1,079 array of java/lang/Object R 0x114c2318 256 - 43,592 659 class java/util/TimeZoneData A 0x00a83a48 56 - 40,488 1,026 java/util/HashMap R 0x14353a18 256 - 37,368 851 class java/beans/Introspector R 0x02fd0070 32 - 30,376 2 java/io/StringWriter R 0x030ed600 32 - 28,560 1 java/lang/String R 0x038604c0 28,528 - 28,528 0 primitive array R 0x114c8f18 256 - 28,120 604 class java/net/URLConnection R 0x030f8730 40 - 23,312 5 org/apache/soap/transport/TransportMessage R 0x038c8020 22,960 - 22,960 0 primitive array R 0x03854648 22,960 - 22,960 0 primitive array A 0x009236c0 32 - 22,408 703 java/lang/ref/Finalizer R 0x02b50dd8 24 - 17,296 1 java/lang/StringBuffer R 0x02cfdff8 24 - 17,256 1 java/lang/StringBuffer (27051 matches but only displayed up to 25.) Matched objects : 27,051 / 639,922 Total Size : 1,882,320 / 36,790,984 Total Subtree Size : 36,790,984 Total Descendants : 612,871 Enter: o[a,s,n,t,d], t[s,c], g[s,c], i, p, d, x or help for more info >

In this example, you entered ot. You accepted the default values except for types to show, which you modified from R,N,A to R,A. This means that you want to see Root and Artificial Root objects only. (Note: An Artificial Root is not a root object in the strictest sense, since another object has a reference to it, but one which HeapRoots detects as being the holder of an object tree, nonetheless. It may be, for example, that it is

only referred to by a backward pointer from one of its own children.) This results in a list of the roots of the 25 largest object trees. In this example you can see that the array of com/ibm/ws/classloader/ReloadableClassLoader$CacheEntry that you saw in the initial dump is indeed far and away the root of the biggest object tree. To dig further into this object tree only, you can use the d command again, but this time, only show the descendants of a single root:

Enter: o[a,s,n,t,d], t[s,c], g[s,c], i, p, d, x or help for more info > d Enter threshold [1048576] Enter max depth or -ve for unlimited [64] Enter 0x to dump from one address or any value for all roots [0x0094fc60] 0x0094fc60

However, you may find that even dumping one object tree down to its leaves presents too much information to be useful. To see quickly which sub-trees are responsible for the greatest amount of memory limit the depth (as here, to 4):

Enter: o[a,s,n,t,d], t[s,c], g[s,c], i, p, d, x or help for more info > d Enter threshold [1048576] Enter max depth or -ve for unlimited [64] 4 Enter 0x to dump from one address or any value for all roots [0x0094fc60] 0x0094fc60 threshold is 1048576 bytes max depth is 4 levels Dumping object at 0x0094fc60 0x0094fc60 [28,612,016/507,751] array of com/ibm/ws/classloader/ReloadableClassLoader$CacheEntry 6 children smaller than 1,048,576 total size/desc: 336/6 0x01fd6fe8 [28,611,968/507,750] com/ibm/ws/classloader/ReloadableClassLoader$CacheEntry 2 children smaller than 1,048,576 total size/desc: 184/1 0x034a9b10 [28,611,936/507,749] com/ibm/ws/classloader/JarClassLoader 10 children smaller than 1,048,576 total size/desc: 4,184/61 0x034a9a10 [28,607,696/507,679] com/ibm/ws/classloader/CompoundClassLoader 15 children smaller than 1,048,576 total size/desc: 263,632/3,087 0x014bfdb0 [28,343,952/504,577] com/ibm/ws/classloader/ExtJarClassLoader 15 children smaller than 1,048,576 total size/desc: 17,280/90 0x014bfdb0 [28,343,952/504,577] com/ibm/ws/classloader/ExtJarClassLoader 0x014bfdb0 [28,343,952/504,577] com/ibm/ws/classloader/ExtJarClassLoader 0x014bfdb0 [28,343,952/504,577] com/ibm/ws/classloader/ExtJarClassLoader

From this dump, you can see that the majority of memory appears to belong to an instance of class com/ibm/ws/classloader/ExtJarClassLoader at address 0x14bfdb0. The fact that it appears multiple times and at different levels presumably means that there are multiple references to the same object. At this point you can further investigate the heap by running the d command again, this time starting with the ExtJarClassLoader object's address:

Enter: o[a,s,n,t,d], t[s,c], g[s,c], i, p, d, x or help for more info > d Enter threshold [1048576] Enter max depth or -ve for unlimited [4] Enter 0x to dump from one address or any value for all roots [0x0094fc60] 0x14bfdb0 threshold is 1048576 bytes max depth is 4 levels Dumping object at 0x014bfdb0

(Root is 0x0094fc60) 0x014bfdb0 com/ibm/ws/classloader/ExtJarClassLoader 15 children smaller than 1,048,576 total size/desc: 17,280/90 0x00911818 com/ibm/ws/classloader/ProtectionClassLoader 6 children smaller than 1,048,576 total size/desc: 568/6 0x00913fc0 com/ibm/ws/bootstrap/ExtClassLoader 14 children smaller than 1,048,576 total size/desc: 618,448/11,130 0x00ba0e18 java/util/Vector 0x03d82440 array of java/lang/Object 5948 children smaller than 1,048,576 total size/desc: 5,321,216/69,051 0x00913fc0 com/ibm/ws/bootstrap/ExtClassLoader 0x00911818 com/ibm/ws/classloader/ProtectionClassLoader

There is an array of Objects at 0x03d82440. It is holding on to 5948 small objects -- smaller than 1 MB -- that account for 5 MB of memory - almost one fifth of the heap.

What would it take for this object array to get garbage-collected? You can use the i command, or single object dump, to see a list of everyone holding references to this object, as well references this object has to others. Because of the way that HeapRoots displays output, listing the parent address after all the child addresses, you need to list all the references, or at least the last few, to make sure that you see the parents listed. A range of n- means "from line n forward." You knew frhom a previous command that this object had 5950 children and one parent, so here you are asking to see the last six children plus the parent:

Enter: o[a,s,n,t,d], t[s,c], g[s,c], i, p, d, x or help for more info > i Enter 0x for object to show info on [NONE] 0x03d82440 Enter range of lines to print in format 'M','L-U','-U' or 'L-' [5940-] 5945- (Displaying from match 5945.) REFERENCES FROM / CHILDREN of 0x03d82440 Addr. Size Name ------------------------------------------------------------------ 0x0090f510 256 class com/ibm/ws/exception/RuntimeWarning 0x0090f610 256 class com/ibm/ws/exception/ConfigurationWarning 0x0090f910 256 class com/ibm/ws/exception/ConfigurationError 0x0090f810 256 class com/ibm/ws/exception/WsException 0x0090f710 256 class com/ibm/ws/exception/WsNestedException 0x0090fa10 256 class com/ibm/ws/runtime/WsServer REFERENCES TO / PARENTS of 0x03d82440 Addr. Size Name ------------------------------------------------------------------ 0x00ba0e18 32 java/util/Vector Total refs : 5,951 Parents, Children : 1 , 5,950 Root Type : N Root-Owner : 0x0094fc60 Total size : 27,843,448 Descendants : 494,139 Size : 40,976 / 36,790,984

To see where the rest of the memory is going, you can dive further into the tree by running the d command against this address, repeating the process until you see another single object (or array) that accounts for a major chunk of memory--in other words, where the amount of the total size for an object drops dramatically beyond that object's own level.

You may not be able to discover a single object or array that accounts for a large amount of memory. For example, a memory leak may be caused by allocation of the same kind of object, over time, under different parents. This means that the leaking, non-garbage-collected objects are scattered throughout the JVM heap.

To look for this kind of leak, use the ts and tc commands to see the which kinds of objects, taken together, are accounting for the most amount of memory and the largest number of objects, respectively:

> ts Enter name to filter on or '-' for no filterting [-] Enter range of lines to print in format 'M','L-U','-U' or 'L-' [1-25] Approximate matches ... Count Size Name ------------------------------------------------------------------ 115,856 14,669,960 primitive array 115,806 3,705,792 java/lang/String 71,704 2,294,528 java/util/HashMap$Entry 19,608 1,739,088 array of java/util/HashMap$Entry 25,508 1,459,288 array of java/lang/Object 34,286 1,097,152 java/util/Hashtable$Entry 39,387 945,288 com/ibm/ejs/util/Bucket 15,835 886,760 java/util/HashMap 14,156 679,488 com/ibm/etools/emf/ref/impl/RefBaseObjectHelperImpl 5,694 545,312 array of java/util/Hashtable$Entry 5,506 220,240 com/ibm/etools/emf/ref/impl/FastOwnedListImpl 9,137 219,288 java/util/ArrayList 4,344 208,512 java/util/Hashtable 6,164 197,248 com/ibm/ejs/util/cache/Bucket 10,707 171,312 java/lang/Integer 9 157,728 array of com/ibm/ejs/util/Bucket 2,278 145,792 org/apache/struts/util/FastHashMap 2,180 139,520 com/ibm/websphere/pmi/stat/CountStatisticImpl 5,523 132,552 java/util/jar/Attributes$Name 33 132,424 array of com/ibm/disthub/impl/util/FastHashtableEntry 3,043 121,720 com/ibm/etools/emf/ecore/impl/ENamedElementImpl 1,233 118,368 com/ibm/websphere/pmi/stat/TimeStatisticImpl 7,232 115,712 java/lang/Object 852 102,240 com/ibm/etools/emf/ecore/impl/EAttributeImpl 2,438 84,216 array of java/lang/String (10864 matches but only displayed up to 25.) Matched types : 10,864 / 10,864 Usage count : 639,922 / 639,922 Total size : 36,790,984 / 36,790,984

If a memory leak is occurring, you will often see one object type that accounts for the vast majority of the number of objects, memory, or both. If such an object's type is a Java or WebSphere base class, you may need help from WebSphere technical support to understand the source of the problem. However, if the leaking object is an application's class, it is likely that the leak is caused by application code. In this case, after identifying the object's class, you can go back through the object tree produced by the d command to find object instances, then use the i command to find all of its parents, as discussed above. At that point, you will still need to contact your application's developers to have them determine and correct the source of the leak in their code, but this information will give them a big head start.

An example of using HeapRoots to diagnose a memory leak

Let's take a look at what HeapRoots tells us in the case of an actual memory leak happening within WebSphere. In this scenario, memory usage sharply and steadily rises when a specific JSP is called. Even after the JSP request returns or is stopped, the memory usage stays at a high level, and is not relieved by garbage collection -- a clear indication of a memory leak.

Trigger a heapdump while memory usage is high, and then use HeapRoots in the interactive mode to diagnose the problem. Let's show objects starting with the roots to an arbitrary depth of 5:

Enter: o[a,s,n,t,d], t[s,c], g[s,c], i, p, d, x or help for more info

> d Enter threshold [1048576] Enter max depth or -ve for unlimited [5] Enter 0x to dump from one address or any value for all roots [NONE] threshold is 1048576 bytes max depth is 5 levels Dumping roots 0x00565d18 [8,903,424/154,290] class java/lang/System 4 children smaller than 1,048,576 total size/desc: 67,560/293 0x007c6218 [8,873,032/154,268] com/ibm/ejs/security/SecurityManager 0x006941d8 [8,873,008/154,267] java/lang/Thread 3 children smaller than 1,048,576 total size/desc: 104/0 0x007c0a48 [8,872,888/154,265] java/util/HashMap 0x007c0a08 [8,872,832/154,264] array of java/util/HashMap$Entry 3 children smaller than 1,048,576 total size/desc: 1,256/21 0x010d7170 [8,871,512/154,239] java/util/HashMap$Entry 1 children smaller than 1,048,576 total size/desc: 40/1 0x00693fc0 [7,630,200/118,950] com/ibm/ws/bootstrap/ExtClassLoader 14 children smaller than 1,048,576 total size/desc: 1,526,848/30,723 0x009440b0 [6,112,512/88,215] java/util/Vector 0x0151f648 [6,112,480/88,214] array of java/lang/Object 3925 children smaller than 1,048,576 total size/desc: 3,679,208/39,002 0x0069f880 [2,024,640/34,702] java/lang/ThreadGroup 3 children smaller than 1,048,576 total size/desc: 289,088/5,091 0x00edc638 [1,735,496/29,607] array of java/lang/ThreadGroup 0x00edc658 [1,735,464/29,606] java/lang/ThreadGroup 2 children smaller than 1,048,576 total size/desc: 14,344/275 0x016283b0 [514,060,576/1,285] array of [LMemEater; 1285 children smaller than 1,048,576 total size/desc: 514,020,560/0

You can see that there are only two root objects, the System class and an array of an array of something called MemEater objects. (Don't expect class names to be this descriptive!) The total tree size of the MemEater array is many times larger than the System class and its descendants. Interestingly, the MemEater array object has no large (greater than 1 Mbytes) descendants; it appears that the sheer number of its children accounts for its size.

Let's look at memory another way: what are the largest objects? Let's use the os option to look at the ten largest objects, not counting their descendants:

Enter: o[a,s,n,t,d], t[s,c], g[s,c], i, p, d, x or help for more info > os Enter name to filter on or '-' for no filterting [-] Enter combination of types to show, R for Roots, A for Artificial Roots, N for Non-Roots [R,N,A] Enter address range in format 'M','L-U','-U' or 'L-' [0x00000000-0xfffffff8] Enter range of lines to print in format 'M','L-U','-U' or 'L-' [10-] -10 Addr. Size Root-Owner Subtree Size Descend. Name --------------------------------------------------------------------------------- N 0x0295d520 400,016 0x016283b0 400,016 0 array of MemEater N 0x204b1e00 400,016 0x016283b0 400,016 0 array of MemEater N 0x029befb0 400,016 0x016283b0 400,016 0 array of MemEater N 0x0e9a20a0 400,016 0x016283b0 400,016 0 array of MemEater N 0x0ea03b30 400,016 0x016283b0 400,016 0 array of MemEater N 0x0f9a7c40 400,016 0x016283b0 400,016 0 array of MemEater N 0x0fa096d0 400,016 0x016283b0 400,016 0 array of MemEater N 0x0fe9d590 400,016 0x016283b0 400,016 0 array of MemEater N 0x2038ce50 400,016 0x016283b0 400,016 0 array of MemEater N 0x0feff020 400,016 0x016283b0 400,016 0 array of MemEater (185166 matches but only displayed up to 10.)

Matched objects : 185,166 / 185,166 Total Size : 524,926,312 / 524,926,312 Total Subtree Size : 1,674,611,560 Total Descendants : 10,982,256

So the ten largest single objects are arrays of MemEater objects, and you can see from the address in the Root-Owner column that they do indeed belong to the monster array of arrays you saw earlier.

So now you have a strong hint that an array of arrays of MemEater objects is causing most of your memory to be consumed. Now you have to take the next step, which HeapRoots cannot do for us -- determine what Java code is causing this to happen. In this case, you know that the memory grows steeply when you access a certain JSP file. Looking at the jsp, you see the following code:

Very Simple JSP Bad JSP2

If you happen to have the source code for MemEater.java, you can look at the MemEater.eatMem() method:

public class MemEater { static String leakString= " .,,; \n" + " LjLfji \n" + " E;,itGDt \n" + " :;tttt... iLDDLjLGiiii \n" + " tGjGtifLfLLtit; iiDLjiijffGi \n" + " GjtjiGGDLjjjffGDi iGDfftijfDDii \n" + " iEEDKEi,ifffjfLLLGtt iiDLftiijLDDi \n" + " .; .,tffjtfLLDti iDDDt,,;jGDi \n" + " .;Dt ..:tGGjjjfLDEti DDi,;;i;Gi \n" + " :L;L;. iDGjjttjDEt KDj,;fj.iK \n" + " ,L Gt. tEGfttjfDEj KtLGjtjfLK \n" + " ,jff,. t;fftjj;t ELLfGGLftD \n" + " ... ttGjtjfGi EWKDEEEDDD \n" + " tDGftfDEWKi;;ijDi \n" + " ijtDKWGjtLL;,;,jEiK \n" + " KDt,,;;jGDj,;;,jEiK \n" + " tGfjjti;i#Kt,;;iLjKt \n" + " tELttjjDEDfjLLLGfGEt \n" + " :L: ttELjtfEfD#WjWt;;tLDt \n" + " .jGf: ttELftfKKiWt,;,::tGDt \n" + " ji if ttLLtfWf,;:,:,ifE \n" + " t;,ij. ttLGGGfjjfjjGt \n" + " .,t;. ttEGEEt \n"; static MemEater[][] ra; String member; public MemEater(String st) { this.member=new String(leakString); } public static void eatMem () {

ra = new MemEater[10000][100000]; for (int i=0; i < = 10000;i++) { for (int j=0; j < = 100000;j++) { ra[i][j]=new MemEater(leakString); } } } }

This is a much simplified version of the kind of code that can cause a memory leak. You can see the method called by the JSP, eatMem, will create a two-dimensional array containing 10000*100000 MemEater objects, each with its own copy of a long string. And the array is a class variable -- it will stay in memory until the MemEater class is garbage-collected, if ever!

Summary of HeapRoots

The HeapRoots tool constructs a snapshot of a JVM's memory space, builds a tree representing the relationship of objects in memory, and gives you various ways of viewing that information to aid you in determining which objects are not being freed. It works on dumps created by the IBM_HEAPDUMP environment variable, which is only recognized by IBM-brand JVMs. Unlike heapdumps created by the standard Hprof-type heapdump, IBM heapdumps store no information about where in Java code objects are being allocated. It is up to you and your application developer to make that connection. The advantage is that it takes very little overhead to enable IBM heapdumps, so it is practical for diagnosing problems in performance-sensitive production environments. A good approach in a production environment would be to try to diagnose a problem using the IBM heapdump approach, and then enable Hprof dumps if you are unable to determine the problem's cause.

Analyzing IBM dumps interactively with HeapWizard

The HeapWizard tool provides a convenient GUI for navigating and interpreting IBM heapdumps. Like Heaproots, HeapWizard is a post-processor application that works upon dumps created by the IBM_HEAPDUMP option of IBM JVMs. It does not work with Hprof heapdumps. HeapWizard is available at ftp://ftp.software.ibm.com/software/websphere/info/tools/heapwizard/HeapWizard.jar.

HeapWizard.jar is an executable jar file. To start HeapWizard, use the following command:

java -Xms128M -Xmx512M -jar HeapWizard.jar

Using HeapWizard

HeapWizard reads IBM heapdump files created using the IBM_HEAPDUMP environment variable. Once the HeapWizard application starts, select File => Open and browse to an IBM heapdump file. Once you have opened the file, you will see a window containing a log of summary information created by HeapWizard as it constructs an object tree (see Figure 2).

Figure 2: Heap Wizard analysis summary

Close this window and bring focus back to the main HeapWizard pane. Double-click the HeapDump icon to see the Classes by Size and Objects by Size views (see Figure 3).

Figure 3: The main panel, tree view

The Classes by Size tree lists the classes whose instances are responsible for occupying most of the memory. The classes are sorted in descending order by the total size, including descendants, of root objects of that class. In the following example, the class Java.lang.String comprises 44,352 total objects in memory, which along with their children occupy 5,441,832 bytes of memory. Of those 44,352 objects, the total cumulative amount of memory for those String objects that are roots is 2,158,000 bytes.

Some entries begin with class, having count 1. These entries represent memory held by actual class objects themselves, not their instances (see Figure 4).

Figure 4: The classes by size view

As you already observed, in the case of a memory leak, the leak will often be evidenced by a single class whose instances far outnumber the next smaller one.

The Objects by Size view lists individual objects in order by the amount of memory, including descendants, that they occupy. In the following example, a single instance of java.lang.ref.Finalizer$FinalizeThread occupies, with its children, 1,903,272 bytes. By itself the object uses a mere 72 bytes (see Figure. 5).

Figure 5: The object tree view

Double-clicking an object in the tree displays a sub-tree of its immediate children, again listed in order by total size. Double-clicking the "fattest" child and grandchild in this tree, for example, produces Figure 6:

Figure 6: Expanding an object in the heap tree

In this example, you can see that the the FinalizerThread object's largest child is an instance of com.ibm.CORBA.iiop.ClientDelegate. Almost all of the ClientDelegate object's memory is in turn held by an attribute of type com.ibm.CORBA.iiop.ORB. Each of these objects is by itself quite small. You could keep on expanding the biggest children until you see a single object that itself is very large, or that is an immediate parent of a large collection of objects.

HeapWizard also provides a command line interface. The readme file contained in the HeapWizard jar file lists the invocation syntax and provides some examples.

An example of heap analysis

So what should you expect to see in the case of an actual leak? Taking the same scenario as in our earlier discussion of the HeapRoots tool, generating an IBM heapdump, and opening it with HeapWizard, you see the Class tree shown in Figure 7.

Figure 7: Class tree

A single root object, the MemEater class, is the parent of the greatest amount of memory by far. This makes sense. Remember that its eatMem() method stores variables in a static attribute -- one that will not be removed unless and until the class itself is removed from memory.

If you look at the Objects tree, you again see that the MemEater class object is the biggest single holder of memory, although by itself it occupies only 256 bytes of memory. If you expand the MemEater class object, you only see two elements: a String object (not shown) occupying a total of 32 bytes, and an array of 1000-element arrays of MemEater instances -- the two-dimensional static array you saw earlier. This two dimensional array occupies about 52 megabytes of memory -- almost all of the memory in our bloated MemEater class! You can see that each of its elements occupies about 53 kilobytes of memory (see Figure 8).

Figure 8: Expanding the largest object in the object by size view

HeapWizard Summary

HeapWizard and HeapRoots are alternative tools for analyzing an IBM heapdump to determine the source of a memory leak. Some users may prefer the interactive GUI provided by HeapWizard. Both tools have the advantage of using as input the lightweight, non-performance-impacting IBM heapdump; both share the disadvantages of not working with standard "Hprof" dumps, and of not associating leaking objects to Java code, since that information is not stored in IBM heapdumps. As with HeapRoots, a reasonable approach would be to try diagnosing a memory leak first by generating an IBM heapdump, then analyze it using HeapWizard. If that is unsuccessful, a next step would be to enable and generate Hprof dumps, and them analyze them manually or with HAT, as described above.

Conclusion Memory leaks are among the most difficult Java application problems to resolve. This is because the symptoms are varied and difficult to reliably reproduce. However, we have outlined a step-by-step approach that will help you to discover memory leaks and pinpoint their sources. Our aim is to make fixing memory leaks more of a science and less of an art.

Part 1 of this paper has concentrated on techniques for investigating memory leaks in WebSphere Application Server for AIX and Windows. Part 2 will focus on the z/OS platform.

About the authors The authors of this paper are:

Steve Eaton (IBM Austin, TX) Steve Eaton has been part of the WebSphere Application Server technical support team for three years.

Frederic Mora (IBM Poughkeepsie, NY) Frederic has been involved in development for ten years and in WebSphere testing for three years. He is now providing support to WebSphere on zSeries customers.

Hany Salem (IBM Austin, TX) Hany Salem is the lead serviceability architect for WebSphere Application Server.

Acknowledgments

This paper benefited greatly from the help of several people. The authors would like to extend their thanks to:

Michel Betancourt (IBM Raleigh, NC) Michel graduated from Florida International University two years ago and has been supporting WebSphere Application Server ever since.

Jim Cunningham (IBM Poughkeepsie, NY) Jim is a performance analyst working on WebSphere for zOS. He has worked on WebSphere performance for the past five years.

Phillip Helm (IBM Raleigh, NC) Phil is the team lead for WebSphere Application Server for z/OS Level 2. He has been in support and service for IBM HTTP Server and WebSphere Application Server for z/OS for 4 years.

Keith Kopycinski (IBM Poughkeepsie, NY) Keith comes from WebSphere Development. He is a technical lead on the WebSphere Level 2 support team and is responsible for identifying and resolving serviceability issues for WebSphere on z/OS.

Arun Kumar (IBM Austin, TX) Arun has been with WebSphere Development and Service organization for a number of years. He works on enhancing Problem Determination and Serviceability characteristics of WebSphere.

David Screen (IBM Hursley, UK) Dave joined IBM in the Java Technology Centre Service team about 18 months ago. He currently works in the Process Automation (Build) team. He develops HeapRoots in his free time because getting to program some Java is fun.

Ron Verbruggen (IBM Raleigh, NC) Ron is a WebSphere Senior Software Engineer. He specializes in WebSphere Serviceability and Support.