I am running a web application on WebSphere Application Server 7.0.0.37 that uses IBM's DynaCache to store data for dropdown lists. When I add load to the code that iterates the list, I end up seeing threads blocking from an analysis of the heap.
Here is the code that retrieves the DistributedMap. Note, I'm not caching the InitialContext as IBM noted it gets cached for you; however, I will be testing with it still to see if it fixes the issue I'm having.
public PinqCache() {
if (getProviderTypeSpecialtyMap() != null) {
getProviderTypeSpecialtyMap().clear();
}
}
public DistributedMap getProviderTypeSpecialtyMap() {
return getDistributedMapByName("/services/cache/provTypeSpecialty");
}
private DistributedMap getDistributedMapByName(String name) {
InitialContext ic;
DistributedMap dm = null;
try {
ic = new InitialContext();
dm = (DistributedMap) ic.lookup(name);
} catch (NamingException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return dm;
}
When I view the thread analysis from IBM's thread analysis tool, I see that 41 threads are being blocked. Here is a stack trace from one of the blocked threads:
Thread Name = WebContainer : 3758
State = Blocked
Monitor = Waiting for Monitor Lock on com/ibm/ws/naming/jcache/Cache@0x0000000041FDBCC0/0x0000000041FDBCCC
Java Stack =
at com/ibm/ws/naming/jcache/Cache.bind(Cache.java:374(Compiled Code))
at com/ibm/ws/naming/jndicos/CNContextImpl.processResolveResults(CNContextImpl.java:3738(Compiled Code))
at com/ibm/ws/naming/jndicos/CNContextImpl.doLookup(CNContextImpl.java:1861(Compiled Code))
at com/ibm/ws/naming/jndicos/CNContextImpl.doLookup(CNContextImpl.java:1762(Compiled Code))
at com/ibm/ws/naming/jndicos/CNContextImpl.lookupExt(CNContextImpl.java:1513(Compiled Code))
at com/ibm/ws/naming/jndicos/CNContextImpl.lookup(CNContextImpl.java:645(Compiled Code))
at com/ibm/ws/naming/util/WsnInitCtx.lookup(WsnInitCtx.java:166(Compiled Code))
at com/ibm/ws/naming/util/WsnInitCtx.lookup(WsnInitCtx.java:180(Compiled Code))
at javax/naming/InitialContext.lookup(InitialContext.java:436(Compiled Code))
at PinqCache.getDistributedMapByName(PinqCache.java:32(Compiled Code))
There are different API methods that are showing blocks:
com/ibm/ws/naming/jcache/Cache.lookup
com/ibm/ws/naming/jcache/Cache.bind
com/ibm/ws/naming/jcache/Cache.setLifetimeProperties
My question is, what would cause this situation where threads are blocking when calling the InitialContext.lookup?
--- UPDATE ---
I've added the thread that owns the monitor lock.
Thread Name = WebContainer : 3709
State = Waiting on Condition
Monitor = Owns Monitor Lock on com/ibm/ws/naming/jcache/Cache@0x0000000041FDBCC0/0x0000000041FDBCCC
Java Stack =
at com/ibm/ws/naming/jcache/Cache.getParentContext(Cache.java:1003(Compiled Code))
at com/ibm/ws/naming/jcache/Cache.bind(Cache.java:382(Compiled Code))
at com/ibm/ws/naming/jndicos/CNContextImpl.processResolveResults(CNContextImpl.java:3738(Compiled Code))
at com/ibm/ws/naming/jndicos/CNContextImpl.doLookup(CNContextImpl.java:1861(Compiled Code))
at com/ibm/ws/naming/jndicos/CNContextImpl.doLookup(CNContextImpl.java:1762(Compiled Code))
at com/ibm/ws/naming/jndicos/CNContextImpl.lookupExt(CNContextImpl.java:1513(Compiled Code))
at com/ibm/ws/naming/jndicos/CNContextImpl.lookup(CNContextImpl.java:645(Compiled Code))
at com/ibm/ws/naming/util/WsnInitCtx.lookup(WsnInitCtx.java:166(Compiled Code))
at com/ibm/ws/naming/util/WsnInitCtx.lookup(WsnInitCtx.java:180(Compiled Code))
at javax/naming/InitialContext.lookup(InitialContext.java:436(Compiled Code))
at com/hcsc/rtb/fs/DAO/pinqDs/PinqCache.getDistributedMapByName(PinqCache.java:32(Compiled Code))
--- UPDATE 2 ---
I was able to capture information from the JNDI trace. I've included some of it that I felt was relevant; however, there is obviously a lot more in here.
I'm consistently seeing the exception below. I've had to *** out the cluster name for privacy.
[10/12/16 14:22:59:986 CDT] 00000084 CNContextImpl > getNameInNamespace Entry
[10/12/16 14:22:59:986 CDT] 00000084 CNContextImpl < getNameInNamespace Exit ********
[10/12/16 14:22:59:986 CDT] 00000084 NameSpace 3 lookupInternal
lookupInternal, nsID=10, name=: this=com.ibm.ws.naming.ipbase.NameSpace@1c781c78[_nameSpaceID=10, _nameSpaceType=2, _contextsTable=java.util.HashMap@a5c8e52d[size=9], _bindingsTable=java.util.HashMap@7cd83d3c[size=24]], currContext=com.ibm.ws.naming.jndicos.CNContextImpl@27442744[contextID=t-shrdm-was70-2Cell01/clusters/rtb_uat2, nameInNamespace=********]
[10/12/16 14:22:59:986 CDT] 00000084 NameSpace 3 validateName
Name is null or empty.
[10/12/16 14:22:59:987 CDT] 00000084 Cache 3 getParentContext
javax.naming.InvalidNameException: A null or empty name is not allowed.
at com.ibm.ws.naming.ipbase.NameSpace.validateName(NameSpace.java:1632)
at com.ibm.ws.naming.ipbase.NameSpace.lookupInternal(NameSpace.java:1115)
at com.ibm.ws.naming.ipbase.NameSpace.intermediateCacheLookup(NameSpace.java:1101)
at com.ibm.ws.naming.jcache.Cache.getParentContext(Cache.java:943)
at com.ibm.ws.naming.jcache.Cache.bind(Cache.java:382)
at com.ibm.ws.naming.jndicos.CNContextImpl.processResolveResults(CNContextImpl.java:3738)
at com.ibm.ws.naming.jndicos.CNContextImpl.doLookup(CNContextImpl.java:1861)
at com.ibm.ws.naming.jndicos.CNContextImpl.doLookup(CNContextImpl.java:1762)
at com.ibm.ws.naming.jndicos.CNContextImpl.lookupExt(CNContextImpl.java:1513)
at com.ibm.ws.naming.jndicos.CNContextImpl.lookup(CNContextImpl.java:645)
at com.ibm.ws.naming.util.WsnInitCtx.lookup(WsnInitCtx.java:166)
at com.ibm.ws.naming.util.WsnInitCtx.lookup(WsnInitCtx.java:180)
at javax.naming.InitialContext.lookup(InitialContext.java:436)
