Spring Batch Partitioning With Permanent Queue As Reply Channel

Question

This issue was initially discussed here:

Partitioned jobs using reply destination

However, I want to clarify what I am seeing. First of all, this is for a system with 40+ partitioned jobs operating on a multiple server cluster. The configuration of the inbound gateway now is:

<int-jms:inbound-gateway id="springbatch.inbound.gateway" 
  connection-factory="springbatch.jmsConnectionFactory" 
  request-channel="springbatch.slave.jms.request" 
  request-destination="requestsQueue" 
  reply-channel="springbatch.slave.jms.response" 
  concurrent-consumers="2" 
  max-concurrent-consumers="2"/>

The outbound gateway is:

<int-jms:outbound-gateway 
  id="outbound-gateway-1"   
  auto-startup="false"
  connection-factory="jmsConnectionFactory" 
  request-channel="partitioned.jms.requests" 
  request-destination="jms.requestsQueue" 
  reply-channel="partitioned.jms.reply" 
  reply-destination="jms.repliesQueue"
  correlation-key="JMSCorrelationID">
  <int-jms:reply-listener/>        
</int-jms:outbound-gateway>

Then I added a job listener to start and stop the outbound gateway on job start and stop. When I start the system up, the JBoss JMX Console shows that there are 2 consumers on the request queue and no consumers on the response queue as expected.

When the partitioned job is started, the job listener calls start on the gateway and I see it added as a handler. Then another handler is also added ( a BridgeHandler from replyListener??) and the reply queue now show 2 consumers on the reply queue. The messages are sent, processed, response returned, and picked up by the gateway reply listener and the job completes.The listener then calls stop on the gateway and it is removed as a handler. The reply queue still has the one 1 extra consumer.

If I try to run the partitioned job a second time, the listener calls start on the gateway and there are 2 consumers on the reply queue. The messages are sent, processed, response returned. The messages appear in the reply queue but they are never picked up and the job hangs.

So, it appears that the stopping and starting of the outbound gateway may not be enough.

If the extra handler is from the replyListener, then does the job listener also need to stop and start it?
If the extra handler is from something else, how can I control it?
How can I further isolate and resolve the problem?

Thanks

The reply messages are returned from the remote partitions (inbound-gateway) and are sitting in the reply queue. The outbound gateway is not picking up the messages on all executions of a job after the first execution ( which is successful). — Mike Rother, Jan 14 '15 at 12:07
Deleted my answer; I mis-read the code. Thanks @ArtemBilan . starting/stopping the gateway should work; I'll run some tests. — Gary Russell, Jan 14 '15 at 14:19

Gary Russell · Answer 1 · 2015-01-14T14:53:54.120

0

The test in this gist works fine for me.

Can you test it in your environment?

If it succeeds, compare the debug log to that from your real application.

If it fails, please upload a TRACE level log from the test someplace.

Thanks

edited Jan 14 '15 at 14:53

answered Jan 14 '15 at 14:05

Gary Russell

166,535
14
146
179

It's generally better to edit your question if you don't have enough rep to edit my answer. I can't speak to jboss but [this answer](http://stackoverflow.com/questions/21984319/why-defaultmessagelistenercontainer-should-not-use-cachingconnectionfactory/21989895#21989895) explains about the caching CF. – Gary Russell Jan 15 '15 at 22:52
how does SingleConnectionFactory wrappering a app server connection factory differ from directly referencing the app server connection factory via JNDI Lookup? – Mike Rother Jan 16 '15 at 00:26
SingleConnectionFactory is a superclass of CachingConnectionFactory. It caches just the connection. Please see the [javadocs](http://docs.spring.io/spring/docs/current/javadoc-api/) for both. – Gary Russell Jan 16 '15 at 00:43
I was also reading about adding a cache-level to the reply-listener What does the do differently? – Mike Rother Jan 16 '15 at 23:20
You typically wouldn't want to do that; it disables consumer caching within the reply `DefaultListenerContainer` which means it would create a new `Consumer` for each message. See the [javadocs for the container](http://docs.spring.io/spring/docs/current/javadoc-api/index.html?org/springframework/jms/listener/DefaultMessageListenerContainer.html). While we expose most of the reply container attributes, such as`concurrentConsumers`, usually the default settings (1 consumer) is enough because the delivery is very lightweight; all the listener thread does is hand the message over to the caller. – Gary Russell Jan 17 '15 at 13:56

Mike Rother · Answer 2 · 2015-01-15T19:27:40.760

I looked at your test case and noticed a difference in the connection factory construction. My configuration was creating the connection factory by:


        <jee:jndi-lookup id="springbatch.jmsConnectionFactory" jndi-name="${connectionFactoryName}"/>>

When I changed to the configuration from your test case:


    <jee:jndi-lookup id="jboss.jmsConnectionFactory" jndi-name="${connectionFactoryName}"/&gt
    <bean id="springbatch.jmsConnectionFactory"
        class="org.springframework.jms.connection.CachingConnectionFactory"&gt
        <property name="targetConnectionFactory" ref="springbatch.jboss.jmsConnectionFactory"/&gt
        <property name="sessionCacheSize" value="10" /&gt
        <property name="cacheProducers" value="true" /&gt
        <property name="cacheConsumers" value="false" /&gt
    </bean&gt

Now I was seeing the messages processed in the preliminary testing. So I would guess the problem was caching of the consumers. The question becomes what is the correct configuration of the jms connection factory. There are 2 options referencing the jms connection of the app server:

clustered
non-clustered

There are 3 options from Spring:

non-wrappered
SingleConnectionFactory
CachingConnectionFactory

So of these 6 option, what is the best configuration and why?

Spring Batch Partitioning With Permanent Queue As Reply Channel

2 Answers2