Java ClassLoader Issue or Concurrency Error?

Question

After a WebLogic app has been running fine for a few weeks I suddenly get an Exception:

<Oct 25, 2014 9:31:11 PM EDT> <Error> <HTTP> <BEA-101020>
    <[ServletContext@60724164[app:whatever3000 module:whatever3000.war path:
    spec-version:2.5]] Servlet failed with Exception
    java.lang.ExceptionInInitializerError

after which the application is completely defunct with NoClassDefFoundError until the app server is restarted.

The full stack trace shows the origin of the issue is a ConcurrentModificationException in a static initializer.

Specifically the equivalent / minimized code is as follows:

package a;
import b;

public class Whatever {
    void doIt()
    {   
        Password p = new Password();
    }   
}

package b;

public final class Password implements Serializable
{
    private static final int PARAM1 = CommonStuff.someStaticMethod();
    ... 
}

import java.util.Properties;

public class CommonStuff
{
    private static Properties prp = new Properties();

    static {
        CommonStuff.load();
    }   

    public static void load()
    {   
        prp.putAll(System.getProperties()); <---FAIL

This is the origin of the Exception:

java.util.ConcurrentModificationException
        at java.util.Hashtable$Enumerator.next(Hashtable.java:1017)
        at java.util.Hashtable.putAll(Hashtable.java:469)
        at b.CommonStuff.load(CommonStuff.java:55)
        at b.CommonStuff.<clinit>(CommonStuff.java:77)
        at b.Password.<clinit>(Password.java:44)
        at a.doIt(Whatever.java:99)

So it seems at some point during the runtime of the application, WebLogic decided to reload classes from package b but when the static block runs it finds that the Properties object has been modified.

I don't know if it's being called concurrently or if it's being called multiple times. It could be that the Properties object is the original instance created when the app loaded fresh the first time and the reloading of CommonStuff class is trying to call putAll() again.

Would it help if I do:

private static Properties prp = null;

static {
    CommonStuff.prp = new Properties();
    CommonStuff.load();
}

I cannot just try things blindly because it's in a production app at a huge company. So I'm trying to understand where I'm going wrong and how to property initialize these variables while classes are being reloaded in the middle of the night.

Any ideas?

Could this be a WebLogic ClassLoader issue?

score 3 · Accepted Answer · answered Oct 29 '14 at 21:48

Classes/instances cannot access some class's members before this class is initialized. Because of this no one can access newly created prp before static constructor returns. Moving prp initializer inside on static {} block makes no difference. Also "old" prp in the "old" CommonStuff and "new" prp are not connected in any way (because "old" and "new" CommonStuff are different classes for JVM). This all make possibility of concurrent modifications of prp look pretty weird.

I believe the reason is in another place. Notice the first line of the stack trace: exception is thrown by Enumerator of Hashtable. Here is the code of the putAll method (as in JDK 8, probably didn't change for many years):

for (Map.Entry<? extends K, ? extends V> e : t.entrySet())
    put(e.getKey(), e.getValue());

Here is the Enumerator that throws an exception - and it is not prp, it is argument's Enumerator.

So the exception is related not to prp but to Map returned by System.getProperties(). The reason is that iterating over system properties map is not thread safe. It seems that another thread is modifying it at the same time.

You need to initialize prp differently. I think clone() is the simplest way.

This looks like it's probably the cause. Someone else independently had the same diagnosis. I have added clone() and we'll see over the next few weeks if it works. Thanks. — squarewav, Oct 31 '14 at 05:50

Java ClassLoader Issue or Concurrency Error?

1 Answers1