Play Java: Memory leaks where the classloader cannot be garbage collected

http://opensource.atlassian.com/confluence/spring/pages/viewpage.action?pageId=2669

Short description of the problem

When an application is loaded in a container environment, it gets its own ClassLoader. In tomcat this will be a WebAppClassLoader. When the application is undeployed, the container in theory drops all references to that class loader, and expects normal garbage collection to take care of the rest. However, if any object that is loaded by the system or container classloader (StandardClassLoader in tomcat) still has a reference to the application class loader or any object loaded by it, the class loader will not be garbage collected. In that case, all the class objects and every object referenced by a static field in any of the classes will not be garbage collected.
I've run into this problem in the context of a spring/hibernate application running on tomcat, but it's really general to any situation where you have classloaders with a lifecycle.

Problems

There are two main patterns that cause this situation. The first one is where a library loaded by the container has a cache that keeps strong references to objects or classes in the application. The second one is where the application has ThreadLocal data that is attached to a container thread. In tomcat this thread will be part of the thread pool, so it will never be garbage collected.

Known offenders

Bean introspection

The bean introspection code will keep a strong cache, but this can easily be cleared with a call to java.beans.Introspector.flushCaches(). There is a listener in the spring library that will take care of this: org.springframework.web.util.IntrospectorCleanupListener.

IntrospectionUtils

org.apache.tomcat.util.IntrospectionUtils has a cache of objects and their methods that it has been used against. This is a Hashtable called objectMethods that is never cleaned up. It will be used to inspect any exceptions that your application throws. In this table, the class is the key and the method the value. Note that a WeakHashMap will not help here as the value has a reference to the key. I haven't found any good way of getting around this, short of removing the cache and making your own version of the jar.
There is also a copy of IntrospectionUtils in org.apache.commons.modeler.util. It seems to have the same problem.

DriverManager

Any JDBC driver loaded in the application (from the WEB-INF/lib directory) will be registered in the system-wide DriverManager. It will not be unloaded unless you add a listener similar to the example below (written by Guillaume Poirier).

CleanupListener.java

public class CleanupListener implements ServletContextListener { 
  public void contextInitialized(ServletContextEvent event) { 
  } 
  public void contextDestroyed(ServletContextEvent event) { 
    try { 
      Introspector.flushCaches(); 
      for (Enumeration e = DriverManager.getDrivers(); e.hasMoreElements();) { 
        Driver driver = (Driver) e.nextElement(); 
        if (driver.getClass().getClassLoader() == getClass().getClassLoader()) { 
          DriverManager.deregisterDriver(driver);          
        } 
      } 
    } catch (Throwable e) { 
      System.err.println("Failled to cleanup ClassLoader for webapp"); 
      e.printStackTrace(); 
    } 
  } 
}

DOM4J

There used be some data saved in ThreadLocals by DOM4J, but this has been fixed in a later version. 1.6.1 seems to work fine for me.

Mozilla Rhino

See Bugzilla: ThreadLocal in Context prevents class unloading.

KeepAliveCache

sun.net.www.http.KeepAliveCache will keep HTTP 1.1 persisent connections open for reuse. It launches a thread that will close the connection after a timeout. This thread has a strong reference to your classloader (through a ProtectionDomain). The thread will eventually die, but if you are doing rapid redeployment, it could still be a problem. The only known solution is to add "Connection: close" to your HTTP responses.

IdleEvictor

If you are using the commons-dbcp connection pooler, you may be activating the idle evictor thread. Because of a bug, this will sometimes keep running even if your application is unloaded. It can be deactivated by calling the setTimeBetweenEvictionRunsMillis(long) method of the org.apache.commons.pool.impl.GenericObjectPool object with a negative parameter.

Jasper

There is a bug in how Jasper (a JSP compiler) pools taglibs. This has been seen to cause memory leaks when used with hibernate, there is more information here: Bugzilla: The forEach JSTL tag doesn't release items

Sun's -server VM

In some cases (see JDK bug 4957990 below) the VM will not free your class loader even if there are no references to it. Try running with -client.

Suspected offenders

CGLIB

CGLIB used to have some memory leaks problems in its proxy code, but they are supposedly fixed in later versions.

commons-logging

If you load commons-logging in your container, it can cause leaks. See Logging/UndeployMemoryLeak for more information. There is also some information about this in the guide: Classloader and Memory Management

How to find your offenders

As for your application, there are two ways to go. Either start with a minimal test application and add stuff until it doesn't unload properly, or start with a complete applications and fix offenders until it unloads. Either way, a profiler will be essential. There are several alternatives, free and commercial, listed below.
Unfortunately I haven't found a profiler that will make it trivial to find these problems. In most cases I have to go through all the classes that are loaded by my class loader, and look for references to them by Objects loaded by the container class loader. This is very time-consuming work and can be quite frustrating - it's quite common to see that your application loads several thousand classes.
According to baliukas: If you run the JVM in debug note, or you use -Xprof, class objects will not be unloaded.

JProfiler

Commercial profiler that gives you a very good graphical representation of your references, but handles any static field in a class object as a garbage collection root.

YourKit Java Profiler

Commercial profiler. In some cases it is able to automatically find references to your ClassLoader. In other cases, your ClassLoader is annotated as a "Other GC" object, whatever that means.

Eclipse Profiler

Free. I've seen this recommended. http://eclipsecolorer.sourceforge.net/

JConsole

Included with the JDK (in 1.5 at least). Gives you some basic information about memory usage etc.

NetBeans Profiler

Free. I've seen this recommended.

HAT

HAT is used to parse the binary output from HPROF (a small profiler that is included with the JDK). Attila submitted a patch to make HAT work better for problems with classloaders. Hat can be found here: http://hat.dev.java.net/, and the patch at https://hat.dev.java.net/servlets/ReadMsg?list=dev&msgNo=1.

Other discussions

Spring forum: Memory Leak
Spring forum: Memory leaks when redeploying web applications
Spring forum: Memory Leak: Spring does not release my objects
Hibernate forum: OutofMemoryError on webapp redeploy (10 pages)
dom4j-Bugs-1070309: ThreadLocal cache
web app using log4j not garbage collected when shutdown with tomcat manager
(commons-logging) j2ee unit tests added: memory leak demonstrated
spring-devel: Further Investigations into OOM Exceptions on Redeploy

Related bugs

Play Java

Tuesday, May 18, 2010

Memory leaks where the classloader cannot be garbage collected