multi-threaded calls from a client to EJBs on Glassfish

Question

I'm running an operation that involves numerous remote EJB calls to a service hosted on glassfish.

There is high latency between the client and the server, so I'm trying to speed up the operation by multi-threading the requests.

Unfortunately, I don't seem to be gaining any speed improvements by multi-threading the requests - instead - my threaded requests are being queued.

My issue is that I can't figure out why they are being queued.

I have checked the thread pool configuration on the server - and inspected it while it is running - I see a corresponding number of threads on the server to the number of threads I have on the client making EJB calls. Most of those threads are idle, most of the time however - the requests simply aren't getting here.

I feel like my calls are being blocked on the client side - from what I'm seeing on the server - both from looking at the threads in jVisualVM - and from looking at the resource usage on the server (its almost idle).

On the client, I'm creating a complete, new InitialContext object / connection for each thread.

On the client, I see most of my threads being blocked within corba:

    "p: default-threadpool; w: 29" daemon prio=10 tid=0x00007f18b4001000 nid=0x608f in Object.wait() [0x00007f19ae32b000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000000f2ef5388> (a java.util.LinkedList)
    at java.lang.Object.wait(Object.java:503)
    at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.resumeOptimizedReadProcessing(CorbaMessageMediatorImpl.java:791)
    - locked <0x00000000f2ef5388> (a java.util.LinkedList)
    at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.setWorkThenReadOrResumeOptimizedRead(CorbaMessageMediatorImpl.java:869)
    at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.handleInput(CorbaMessageMediatorImpl.java:1326)
    at com.sun.corba.ee.impl.protocol.giopmsgheaders.FragmentMessage_1_2.callback(FragmentMessage_1_2.java:122)
    at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.handleRequest(CorbaMessageMediatorImpl.java:742)
    at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.dispatch(CorbaMessageMediatorImpl.java:539)
    at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.doWork(CorbaMessageMediatorImpl.java:2324)
    at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.performWork(ThreadPoolImpl.java:497)
    at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:540)

my threads are blocked on

"Pool-27" daemon prio=10 tid=0x00007f19b403c800 nid=0x4ef8 waiting on condition [0x00007f19ac70e000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000000f2f66078> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
    at com.sun.corba.ee.impl.transport.CorbaResponseWaitingRoomImpl.waitForResponse(CorbaResponseWaitingRoomImpl.java:173)
    at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.waitForResponse(SocketOrChannelConnectionImpl.java:1021)
    at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.waitForResponse(CorbaMessageMediatorImpl.java:279)
    at com.sun.corba.ee.impl.protocol.CorbaClientRequestDispatcherImpl.marshalingComplete1(CorbaClientRequestDispatcherImpl.java:407)
    at com.sun.corba.ee.impl.protocol.CorbaClientRequestDispatcherImpl.marshalingComplete(CorbaClientRequestDispatcherImpl.java:368)
    at com.sun.corba.ee.impl.protocol.CorbaClientDelegateImpl.invoke(CorbaClientDelegateImpl.java:273)
    at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:200)
    at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:152)
    at com.sun.corba.ee.impl.presentation.rmi.codegen.CodegenStubBase.invoke(CodegenStubBase.java:227)

Something else that bugs me, is no matter how many threads I run on the client - I only get one - sometimes 2 - 'RMI TCP Connection' threads. But I'm not sure if this is involved in EJB remoting or not.

Why / Where are my client threads being blocked on these remote calls?

Note, this isn't an bandwidth issue - the amount of data being transferred is small - but there is high latency. So making more requests in parallel should be giving me an improvement. Instead - my overall time stays about the same - but individual requests go from taking 500 ms to taking 5000 ms, for example, as I increase the client thread count.

Anything above 2 or 3 threads just makes no difference at all to the overall time.

If I take the latency out of the picture, by running the same process local on the same system as the GlassFish server, performance is as expected - too fast to really even debug - so there is no question of the server being able to handle the workload.

I just can't seem to issue requests in parallel from a client.

score 0 · Answer 1 · edited May 23 '17 at 11:58

0

Check of what type your EJBs are. If you call the same Stateful or Singleton EJB, your calls are synchronized - only single call is allowed for them at a time by default, no matter if you call the same method or different ones. With Stateless EJBs this is not the case, but your calls may be blocked if your pool is depleted - check the size of EJB pool on your server and increase it.

With Stateful, you cannot change the behavior (see this answer, but you may instantiate a new EJB for every remote client. With Singleton, you should consider adjusting concurrency management to allow more throughput.

edited May 23 '17 at 11:58

Community

1
1

answered Oct 25 '15 at 13:10

OndroMih

7,280
1
26
44

Hmm, I checked on the bean types - and I did find one that was inadvertently set to a Singleton. Fixing that hasn't had much impact, however, most of the beans that I'm accessing were already set to StatelessSessionBean type. There are a couple of beans in the system that are Singleton's by design - but I assume that if a Stateless bean references a singleton bean - internally, it should only synchronize on those calls that touch the singleton - and not synchronize the entire stateless bean, yes? – user2163960 Oct 25 '15 at 18:10
You are correct - Stateless EJBs are assigned one per each request therefore they do not block other requests. Singleton can only block when its method is running. Maybe your client synchronizes EJB executions for whatever reason - check if it is possible to configure CORBA/RMI thread pool in client. – OndroMih Oct 25 '15 at 18:43

multi-threaded calls from a client to EJBs on Glassfish

1 Answers1