ClassLoader Leaks
-
Upload
mattias-jiderhamn -
Category
Technology
-
view
147 -
download
0
description
Transcript of ClassLoader Leaks
![Page 1: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/1.jpg)
Mattias Jiderhamn – java.jiderhamn.se
Classloader leaks
Mattias Jiderhamn
![Page 2: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/2.jpg)
Mattias Jiderhamn – java.jiderhamn.se
java.lang.OutOfMemoryError: PermGen space
:-(
![Page 3: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/3.jpg)
Mattias Jiderhamn – java.jiderhamn.se
Local dev env
![Page 4: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/4.jpg)
Mattias Jiderhamn – java.jiderhamn.se
Continuous Deploy
![Page 5: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/5.jpg)
Mattias Jiderhamn – java.jiderhamn.se
Java Memory Model
Stack
Heap
PermGen
![Page 6: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/6.jpg)
Mattias Jiderhamn – java.jiderhamn.se
Java Memory ModelPer threadLocal variables and
method parametersStack
![Page 7: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/7.jpg)
Mattias Jiderhamn – java.jiderhamn.se
Java Memory Model
HeapYoung generation Old generation /
tenured spaceEdenspace
Survivorspace
Object instances
![Page 8: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/8.jpg)
Mattias Jiderhamn – java.jiderhamn.se
Java Memory ModelPermanent Generationjava.lang.Class instances
etcNamed before JEE and
class unloadingRenamed Metaspace in Java 8
PermGen
![Page 9: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/9.jpg)
Mattias Jiderhamn – java.jiderhamn.se
java.lang.OutOfMemoryError: PermGen space
= Too many classes are loaded
Java Memory Model
![Page 10: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/10.jpg)
Mattias Jiderhamn – java.jiderhamn.se
Reason for OOME
1. Your application is too large-XX:MaxPermSize=256MJava 8: Metaspace auto increase by default
2. java.lang.Class instances could not be garbage collected after redeploy
java.lang.OutOfMemoryError: PermGen space
![Page 11: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/11.jpg)
Mattias Jiderhamn – java.jiderhamn.se
RedeployApplication Server
ClassLoader
app.war
ClassLoader
app.war’
ClassLoader
app.war’’
![Page 12: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/12.jpg)
Mattias Jiderhamn – java.jiderhamn.se
Reference types• Strong (i.e. normal) reference–Never GC:ed if reachable
• Soft reference–GC:ed before OutOfMemoryError
•Weak reference (WeakHashMap)–GC:ed when no Strong or Soft refs
• Phantom reference–… won’t prevent GC
![Page 13: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/13.jpg)
Mattias Jiderhamn – java.jiderhamn.se
Example
Foo
WeakHashMap
Map m = new WeakHashMap();Foo myFoo = new Foo();m.put(myFoo, ”bar”);myFoo = null;
![Page 14: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/14.jpg)
Mattias Jiderhamn – java.jiderhamn.se
GC reachability
GC roots
![Page 15: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/15.jpg)
Mattias Jiderhamn – java.jiderhamn.se
ClassLoader refsApplication Server
ClassLoader
Class ClassClass
Instance
![Page 16: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/16.jpg)
Mattias Jiderhamn – java.jiderhamn.se
How leaks happenApplication Server
ClassLoader
Class ClassClass
InstanceGC root
![Page 17: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/17.jpg)
Mattias Jiderhamn – java.jiderhamn.se
Application Server• Application Server bugs• Logging frameworks
– Apache Commons LoggingUnless LogFactory.release()
– Log4j - some configsUnless LogManager.shutdown()
– java.util.logging custom Level• Bean Validation API (JSR 303)• Unified Expression Language (javax.el)
![Page 18: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/18.jpg)
Mattias Jiderhamn – java.jiderhamn.se
GC roots• Class loaded by system ClassLoader
– static field in JDK classes (java.* etc)• Live thread
– Stack – local vars, method params–java.lang.Thread instance
• Object held as synchronization monitor• JNI references• JVM specials…
![Page 19: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/19.jpg)
Mattias Jiderhamn – java.jiderhamn.se
System classes•java.sql.DriverManager• Bean introspection cache, shutdown hooks,
custom default Authenticator, custom security Provider, custom MBeans, custom ThreadGroup, custom property editor, …
• Reference to contextClassLoader of first caller
![Page 20: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/20.jpg)
Mattias Jiderhamn – java.jiderhamn.se
DriverManager
![Page 21: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/21.jpg)
Mattias Jiderhamn – java.jiderhamn.se
DriverManagerApplication Server
app.war
mysql-jdbc.jar
JRE
DriverManager
ClassLoader
com.mysql.jdbc.Driver
1) …
… or 2)DriverManager.deregisterDriver(…)
![Page 22: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/22.jpg)
Mattias Jiderhamn – java.jiderhamn.se
Context shutdownpublic class MyCleanupListener implements
javax.servlet.ServletContextListener {
...
/** Called when application is undeployed */
public void contextDestroyed(
ServletContextEvent servletContextEvent) {
DriverManager.deregisterDriver(…);
}
}
![Page 23: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/23.jpg)
Mattias Jiderhamn – java.jiderhamn.se
Threads• Thread stack
– Local variables– Method parameters
• MyThread extends ThreadMyRunnable implements Runnable
• contextClassLoader– Example HTTP 1.1 Keep-Alive-Timer
![Page 24: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/24.jpg)
Mattias Jiderhamn – java.jiderhamn.se
Context shutdownpublic class MyCleanupListener implements
javax.servlet.ServletContextListener {
...
/** Called when application is undeployed */
public void contextDestroyed(
ServletContextEvent servletContextEvent) {
DriverManager.deregisterDriver(…);
// Stop threads here!
}
}
![Page 25: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/25.jpg)
Mattias Jiderhamn – java.jiderhamn.se
Stopping Threadspublic class MyThread extends Thread {
public void run() { while(true) { // Bad idea! // Do something } }
}
![Page 26: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/26.jpg)
Mattias Jiderhamn – java.jiderhamn.se
Stopping Threadspublic class MyThread extends Thread {
private boolean running = true;
public void run() { while(running) { // Until stopped // Do something } }
public void shutdown() { running = false; }}
private volatile boolean running = true;
Heinz Kabutz / Java Specialists’ - The Law of the Blind Spot
![Page 27: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/27.jpg)
Mattias Jiderhamn – java.jiderhamn.se
Threads• Thread stack
– Local variables– Method parameters
• MyThread extends ThreadMyRunnable implements Runnable
• contextClassLoader– Example HTTP 1.1 Keep-Alive-Timer
• ThreadLocal
![Page 28: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/28.jpg)
Mattias Jiderhamn – java.jiderhamn.se
ThreadLocal
WeakHashMap<Thread, ?>
ThreadLocal
Thread Foo
![Page 29: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/29.jpg)
Mattias Jiderhamn – java.jiderhamn.se
ThreadLocalThreadLocal JavaDoc
”Each thread holds an implicit reference to its copy of a thread-local variable as long
as the thread is alive and the ThreadLocal instance is accessible;
…”
![Page 30: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/30.jpg)
Mattias Jiderhamn – java.jiderhamn.se
ThreadLocal
ThreadLocalMapEntry
Thread
ThreadLocal Foo
put()
set()
![Page 31: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/31.jpg)
Mattias Jiderhamn – java.jiderhamn.se
ThreadLocalPooled threads:•Threads may outlive ClassLoader•ThreadLocal → ThreadGlobal!
(?)
![Page 32: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/32.jpg)
Mattias Jiderhamn – java.jiderhamn.se
ThreadLocal
ThreadLocalMapEntry
Thread
ThreadLocal Foo
ClassClassLoader
static
Class
![Page 33: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/33.jpg)
Mattias Jiderhamn – java.jiderhamn.se
ThreadLocal
ThreadLocalMapEntry
Thread
ThreadLocal Foo
ClassClassLoader
Stale entry
![Page 34: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/34.jpg)
Mattias Jiderhamn – java.jiderhamn.se
ThreadLocalThreadLocalMap JavaDoc
”However, since reference queues are not used, stale entries are guaranteed
to be removed only when the table starts running out of space”
![Page 35: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/35.jpg)
Mattias Jiderhamn – java.jiderhamn.se
ThreadLocal•Custom value (incl references)–static ThreadLocal = leak–otherwise = unpredicted GC:ing
•Custom ThreadLocal = no leak
![Page 36: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/36.jpg)
Mattias Jiderhamn – java.jiderhamn.se
ThreadLocal
try { myThreadLocal.set(foo); …}finally { myThreadLocal.remove();}
Always clear your ThreadLocals!
![Page 37: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/37.jpg)
Mattias Jiderhamn – java.jiderhamn.se
Known offendersApache ActiveMQApache AxisApache BatikApache Commons Pool / DBCPApache CXFAWT ToolkitBean Validation API / JSR 303CGLIB (Hibernate / Spring / JBoss /
Apache Geronimo)dom4jEclipseLinkGeoToolsGoogle GuiceGroovyGWT / javax.imageioHessianiCal4JInfinispanIntrospectionUtils
JarURLConnectionJava Advanced Imaging (JAI)JavassistJava Cryptography Architecture (JCA)Java Server Faces 2javax.security.auth.Policy/login.ConfigurationJGroupsLogbackJAXBMozilla RhinoMVELOpenOffice Java Uno RunTime (JURT)Oracle JDBCRMISerialization (<= JDK 1.4.2)Spring frameworkUnified Expression Language / javax.elURLConnection + HTTP 1.1XML parsing (DocumentBuilderFactory)
![Page 38: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/38.jpg)
Mattias Jiderhamn – java.jiderhamn.se
Leak prevention lib• Application server independent• Covers much more than Tomcat
– System class references avoided/cleared– Threads are killed– ThreadLocals are cleared– Known offenders handled
• Logs warnings• Apache 2 licensed
– You may modify and redistribute
![Page 39: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/39.jpg)
Mattias Jiderhamn – java.jiderhamn.se
Leak prevention lib
0 runtime dependencies1 class2 XML snippets
![Page 40: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/40.jpg)
Mattias Jiderhamn – java.jiderhamn.se
Leak prevention libpom.xml
<dependency>
<groupId>se.jiderhamn</groupId>
<artifactId>classloader-leak-prevention</artifactId>
<version>1.9.2</version>
</dependency>
web.xml
<listener>
<listener-class>se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor
</listener-class>
</listener>
![Page 41: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/41.jpg)
Mattias Jiderhamn – java.jiderhamn.se
Tomcat
Bugzilla #48895
ThreadLocalMap
ThreadCleanupThread
remove()
Unsafe!
get()set()
remove()
Removed in 6.0.27
Tomcat 7.0.6+ renews the thread pool whenever an application is redeployed.
Disposable threads for lifecycle events.Bugzilla #49159
![Page 42: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/42.jpg)
Mattias Jiderhamn – java.jiderhamn.se
Leak prevention lib
set(null) ThreadLocalMapEntry
Thread
ThreadLocal Foo
Stale
CleanupThread
![Page 43: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/43.jpg)
Mattias Jiderhamn – java.jiderhamn.se
Bean Validation APIVersion 1.0.0.GA
javax.validation.Validation
Application Servervalidation-api-1.0.0.GA.jar
app.warhibernate-validator.jar
![Page 44: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/44.jpg)
Mattias Jiderhamn – java.jiderhamn.se
javax.el APIjavax.el.BeanELResolver
![Page 45: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/45.jpg)
Mattias Jiderhamn – java.jiderhamn.se
OpenOffice JURTcom.sun.star.lib.util.AsynchronousFinalizer
![Page 46: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/46.jpg)
Mattias Jiderhamn – java.jiderhamn.se
GC choice matters• Parallell GC seems to have problems
collecting java.lang.Class instances• CMS needs extra parameter
-XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled
• G1-XX:+UseG1GC
![Page 47: ClassLoader Leaks](https://reader033.fdocuments.net/reader033/viewer/2022061223/54c6423f4a79596b248b45a4/html5/thumbnails/47.jpg)
Mattias Jiderhamn – java.jiderhamn.se
Linkshttp://java.jiderhamn.se
• Leak prevention library incl sources (GitHub)• Step by step guide: acquire heap
dump and analyze for leaks usingEclipse Memory Analyzer (MAT)
• List of known offenders and links to bug reports• Submit your own reports and patches!
Que?Join the fight!