Friday, May 07, 2010

Application ClassLoader’s Footprint Memory Leak in Shared Containers

Basic Reason for the leak:
Every time an application is deployed and un-deployed in a container, the application is given its own class loader (application class loader) by the container’s class loader. This application class loader might end up creating a cyclic reference to the container’s class loader, which makes it un-garbage collectible. From my observations getting rid of the application’s class loader from memory (1 to 2 MBs) could be a very difficult and tedious task because of the amount of third party tools we use, but can be contained, although not ideal.

Avoiding the leak from Crossing Threshold:
a)      One Main Culprit : Spring Class Based Proxies
Many Enterprise applications use Spring, AOP and Hibernate as tools for their projects. With Hibernate comes the Session Factory beast in memory. The Hibernate Session Factories can become huge based on the project’s DB tables count and Hibernate Mapping Structure (150MB for the application that I work on). So when Session Factory gets attached to the un-garbage collectible Class Loader which has been left behind, we have huge memory leak in containers causing us losing valuable resources.
So how does this Session Factory get attached to the un-garbage collectible class loader?
Spring uses proxies to weave in Advices. Transactions are most commonly “AOP Advised” in Enterprise Applications using Declarative Transaction Management.
As you might know, there are 2 types of Proxies that Spring creates, based on the implementation of the class that needs to be advised
1)      Interface Based Proxies – not a problem
  1. Spring uses JDK proxies if there is an interface available for all the advised methods; this is the default type of proxy that is created. A new class is created with methods that needs to be advised, through which advises will be weaved in. A call is then delegated from the new class to the actual class that had to be advised.
2)      Class Based Proxies – A Problem
  1. Spring uses CGLIB to create proxies if an interface is not available for the existing methods that needs to be advised. This type of Proxy should be an exception in general, and it is only provided for backward compatibility for third party libraries that cannot be modified.
  2. The problem here is CGLIB’s implementation leads it to tie a hard link between the Application Class Loader and the Session Factory when it is trying to advise the method of particular transaction choice.
  3. Hence trying to avoid Class based proxies for Spring Transaction Management, in an application with an existing Class Loader leak, may avoid it from crossing the “Threshold” (2 MB to 154MB in a project that I was working on).
b)      Some Third Party Libraries:
There are some third party tools that do the same thing of attaching itself to the class loader and creating a hard link between the runtime classes that they create and the class loader itself. Some examples that I know of right away are Dozer and Apache Common’s Logging.
So it looks like there is no good way to solve the problem of Application Class Loader’s Footprint leak in a Shared Container turning out of control, hence the intent of this article is to make one aware of the issues and point them in the right direction.

PermGen Space will be affected as well
Now this leak may also cause your PermGen Space run out of memory since the meta-inf about the loaded classes may not be garbage collected.
Krank Keviet has done a pretty good job explaining the issue in one of his blogs with respect to Perm Gen Space, a reference to his blog is below.

J2EE Classloaders Demystified

Classloader leaks: the dreaded "java.lang.OutOfMemoryError: PermGen space" exception


Did you ever encounter a java.lang.OutOfMemoryError: PermGen space error when you redeployed your application to an application server? Did you curse the application server, while restarting the application server, to continue with your work thinking that this is clearly a bug in the application server. Those application server developers should get their act together, shouldn't they? Well, perhaps. But perhaps it's really  your fault!
Take a look at the following example of an innocent looking servlet.

import java.util.logging.*;
import javax.servlet.*;
import javax.servlet.http.*;

public class MyServlet extends HttpServlet {
 protected void doGet(HttpServletRequest request, HttpServletResponse response)
 throws ServletException, IOException {
 // Log at a custom level
 Level customLevel = new Level("OOPS", 555) {};
 Logger.getLogger("test").log(customLevel, "doGet() called");
Try to redeploy this little sample a number of times.  I bet this will eventually fail with the dreaded java.lang.OutOfMemoryError: PermGen space error. If you like to understand what's happening, read on.

The problem in a nutshell

Application servers such as Glassfish allow you to write an application (.ear, .war, etc) and deploy this application with other applications on this application server. Should you feel the need to make a change to your application, you can simply make the change in your source code, compile the source, and redeploy the application without affecting the other still running applications in the application server: you don't need to restart the application server. This mechanism works fine on Glassfish and other application servers (e.g. Java CAPS Integration Server).

The way that this works is that each application is loaded using its own classloader. Simply put, a classloader is a special class that loads .class files from jar files. When you undeploy the application, the classloader is discarded and it and all the classes that it loaded, should be garbage collected sooner or later.

Somehow, something may hold on to the classloader however, and prevent it from being garbage collected. And that's what's causing the java.lang.OutOfMemoryError: PermGen space exception.

PermGen space

What is PermGen space anyways? The memory in the Virtual Machine is divided into a number of regions. One of these regions is PermGen. It's an area of memory that is used to (among other things) load class files. The size of this memory region is fixed, i.e. it does not change when the VM is running. You can specify the size of this region with a commandline switch: -XX:MaxPermSize . The default is 64 Mb on the Sun VMs.

If there's a problem with garbage collecting classes and if you keep loading new classes, the VM will run out of space in that memory region, even if there's plenty of memory available on the heap. Setting the -Xmx parameter will not help: this parameter only specifies the size of the total heap and does not affect the size of the PermGen region.

Garbage collecting and classloaders

When you write something silly like

private void x1() {
        for (;;) {
            List c = new ArrayList();
you're continuously allocating objects; yet the program doesn't run out of memory: the objects that you create are garbage collected thereby freeing up space so that you can allocate another object. An object can only be garbage collected if the object is "unreachable". What this means is that there is no way to access the object from anywhere in the program. If nobody can access the object, there's no point in keeping the object, so it gets garbage collected. Let's take a look at the memory picture of the servlet example. First, let's even further simplify this example:

import javax.servlet.*;
import javax.servlet.http.*;

public class Servlet1 extends HttpServlet {
 private static final String STATICNAME = "Simple";
 protected void doGet(HttpServletRequest request, HttpServletResponse response)
 throws ServletException, IOException {
After loading the above servlet, the following objects are in memory (ofcourse limited to the relevant ones):

In this picture you see the objects loaded by the application classloader in yellow, and the rest in green. You see a simplified container object that holds references to the application classloader that was created just for this application, and to the servlet instance so that the container can invoke the doGet() method on it when a web request comes in. Note that the STATICNAME object is owned by the class object. Other important things to notice:

  1. Like each object, the Servlet1 instance holds a reference to its class (Servlet1.class).
  2. Each class object (e.g. Servlet1.class) holds a reference to the classloader that loaded it.
  3. Each classloader holds references to all the classes that it loaded.
The important consequence of this is that whenever an object outside of AppClassloader holds a reference to an object loaded by AppClassloader, none of the classes can be garbage collected.

To illustrate this, let's see what happens when the application gets undeployed: the Container object nullifies its references to the Servlet1 instance and to the AppClassloader object.

As you can see, none of the objects are reachable, so they all can be garbage collected. Now let's see what happens when we use the original example where we use the Level class:

import java.util.logging.*;
import javax.servlet.*;
import javax.servlet.http.*;

public class LeakServlet extends HttpServlet {
 private static final String STATICNAME = "This leaks!";
 private static final Level CUSTOMLEVEL = new Level("test", 550) {}; // anon class!

 protected void doGet(HttpServletRequest request, HttpServletResponse response)
 throws ServletException, IOException {
 Logger.getLogger("test").log(CUSTOMLEVEL, "doGet called");
Note that the CUSTOMLEVEL's class is an anonymous class. That is necessary because the constructor of Level is protected. Let's take a look at the memory picture of this scenario:

In this picture you see something you may not have expected: the Level class holds a static member to all Level objects that were created. Here's the constructor of the Level class in the JDK:
protected Level(String name, int value) { = name;
 this.value = value;
 synchronized (Level.class) {
Here known is a static ArrayList in the Level class. Now what happens if the application is undeployed?

Only the LeakServlet object can be garbage collected. Because of the reference to the CUSTOMLEVEL object from outside of AppClassloader, the  CUSTOMLEVEL anyonymous class objects (LeakServlet$1.class) cannot be garbage collected, and through that neither can the AppClassloader, and hence none of the classes that the AppClassloader loaded can be garbage collected.
Conclusion: any reference from outside the application to an object in the application of which the class is loaded by the application's classloader will cause a classloader leak.

More sneaky problems

I don't blame you if you didn't see the problem with the Level class: it's sneaky. Last year we had some undeployment problems in our application server. My team, in particular Edward Chou, spent some time to track them all down. Next to the problem with Level, here are some other problems Edward and I encountered. For instance, if you happen to use some of the Apache Commons BeanHelper's code: there's a static cache in that code that refers to Method objects. The Method object holds a reference to the class the Method points to. Not a problem if the Apache Commons code is loaded in your application's classloader. However, you do have a problem if this code is also present in the classpath of the application server because those classes take precedence. As a result now you have references to classes in your application from the application server's classloader... a classloader leak!

I did not mentiond yet the simplest recipe for disaster: a thread started by the application while the thread does not exit after the application is undeployed.

Detection and solution

Classloader leaks are difficult. Detecting if there's such a leak without having to deploy/undeploy a large number of times is difficult. Finding the source of a classloader leak is even trickier. This is because all the profilers that we tried at the time, did not follow links through classloaders. Therefore we resorted to writing some custom code to find the leaks from memory dump files. Since that exercise, new tools came to market in JDK 6. The next blog will outline what the easiest approach today is for tracking down a glassloader leak.  

Spring 2.5 Makes Unit Testing Easy with Annotations


One feature of Spring 2.5 (previously known as Spring 2.1) that hasn’t gotten a lot of mention is its significantly enhanced unit testing framework, including the brand new org.springframework.test.context package. Rod Johnson mentioned it in his updated article on Spring 2.5 on TheServerSide, but other than that, I tried searching for someone blogging about this subject, and failed – so I figured I’d jot down my impressions about this great improvement.

Prior to Spring 2.5, Spring provided a series of superclasses that your unit tests were required to inherit. These superclasses (AbstractDependencyInjectionSpringContextTests and the like) provided dependency injection into unit tests, and autowiring by type as default behavior. For those who are unit testing transactional systems, the AbstractTransactionalDataSourceSpringContextTests base class is essential for setting up transaction management and auto-rollback around each unit test.

Spring 2.5 continues this rich and convenient testing framework, while now removing the requirement that your unit tests extend Spring framework tests. This is a welcome change, as it frees up your unit test hierarchy to allow you more flexibility in inheritance.

Instead of providing functionality through subclassing and very specific method overriding, a series of Java 5 annotations are provided to handle various aspects of test data management.

As usual, the Spring documentation has excellent coverage of this new functionality, starting with its coverage of the test context concept, provided by the @ContextConfiguration annotation, which is used in place of the legacy method override of getConfigLocations.

To wit, where before, including several ApplicationContext XML files required the following:
    protected String[] getConfigLocations() {
    return new String[] { "classpath:applicationContext.xml",
        "classpath:applicationContext-test.xml" };

… with Spring 2.5 all that is required is a simple annotation at the unit test class level:
 @ContextConfiguration(locations = { "/applicationContext.xml",
    "/applicationContext-acegi.xml", "/applicationContext-test.xml" })

The default ContextLoader will automatically find these resources on the classpath.

Similar simplicity is provided in the realm of transactional support and DI. In prior versions of Spring, the superclass hierarchy dictated whether or not transactional and DI support was provided. In Spring 2.5, the default ContextLoader will also auto-wire dependencies in the test class, annotated via the typical @Autowired (by type) or @Resource (by name) methods.

For transaction management, the standard Spring @Transactional annotation is supported, to wrap transactions around unit test methods (and rolling back after method completion, as Spring 2.0 used to do). Spring 2.5 provides a couple additional annotations to further specify transactional behavior within a unit test, including the @NotTransactional annotation (for methods not requiring one), and the @Rollback annotation, which can be used to override the default auto-rollback behavior.

Finally, since you’re no longer extending a base class, you can’t override the onSetUp*Transaction and onTearDown*Transaction methods, like in Matt Raible’s example from way back when (among many others).

Instead, the standard JUnit 4 @Before and @After annotations are used in place of onSetUpInTransaction and onTearDownInTransaction. An obvious benefit is the fact that you can annotate multiple methods with these annotations, instead of the single method that was imposed by the override-based implementation. Spring-specific @BeforeTransaction and @AfterTransaction annotations are used to replace the onSetUpBeforeTransaction and onTearDownAfterTransaction methods.

There’s some additional annotations that allow for test timeout measurement (above and beyond what JUnit 4 provides), test repeating, and a few other goodies.

I tested this on a couple existing Spring projects that I’ve worked on, and the transition process was extremely smooth, and allowed me to greatly simplify the unit testing code by stripping out the stuff that used to be required by the Spring class-based unit testing framework.

Unfortunately, although Spring provides some support for TestNG, out of the box it will still require inheriting a [new] Spring 2.5 class, AbstractTransactionalTestNGSpringContextTests. Hopefully Cedric and Alexandru can work with the Spring team to enhance this support to match (or exceed!) the level of support offered to JUnit 4 users.

Spring 2.5 will be released soon, and I bet a lot of users will be buzzing about the unit test enhancements there. Enjoy!

PropertyPlaceholderConfigurer in Spring 2.5

While not explicitly highlighted in the Spring 2.5 “What’s New” section, this is a nice little bit to remove a few lines of XML. Spring 2.5 (with the introduction of a new “context:” namespace) takes the common use of a PropertyPlaceholderConfigurer externalizing properties and folds it from the awkward 5-line syntax into a single line.

Before (Spring 2.0 and earlier):
<bean id="propertyConfigurer"
 <property name="location">

After (Spring 2.5):
<context:property-placeholder location="file:${user.home}/"/>

The nit-picky will note that you will need to add in the spring-context-2.5.xsd schema reference, however you may be using that already anyway in Spring 2.5 to take advantage of some of the Java-based annotation functionality.