0

I'm using

  1. Spring Batch

    • Step 1
    • Step 2 Master (Partitioner)
    • Step 3

  2. Spring Integration (JMS) to communicate Master and Slave

The issue we are seeing is, the first slave handles all JMS messages instead of even distribution between slaves.

See configuration as below

  1. Master 

    <bean id="PreProcess" class="com.job.tasklet.PreProcessTasklet" scope="step">
    <constructor-arg index="0" value="${run.slave}"/>
    <property name="maxNumberOfSlaves" value="#{jobParameters['max-slave-count']}"/>
</bean>

<bean id="PostProcess" class="com.job.tasklet.PostProcessTasklet" scope="prototype">
    <constructor-arg index="0" ref="chpsJobDataSource"/>
</bean>


<bean id="partitioner" class="com.job.partition.DatabasePartitioner" scope="step">
    <constructor-arg index="3" value="${max.row.count}"/>
</bean>

<bean id="partitionHandler" class="com.job.handler.StepExecutionAggregatorHandler">
    <property name="stepName" value="processAutoHoldSlaveStep"/>
    <property name="gridSize" value="${grid.size}"/>
    <property name="replyChannel" ref="aggregatedGroupRuleReplyChannel"/>
    <property name="messagingOperations">
        <bean class="org.springframework.integration.core.MessagingTemplate">
            <property name="defaultChannel" ref="groupRuleRequestsChannel"/>
        </bean>
    </property>
</bean>


    <!-- Request Start -->
<int:channel id="groupRuleRequestsChannel" />
<int-jms:outbound-channel-adapter channel="groupRuleRequestsChannel" jms-template="jmsTemplateToSlave"/>

<bean id="jmsTemplateToSlave" class="org.springframework.jms.core.JmsTemplate">
    <property name="connectionFactory" ref="connectionFactory"/>
    <property name="receiveTimeout" value="5000"/>
    <property name="defaultDestinationName" value="defaultRequest"/>
</bean>

<bean id="jmsTemplateFromSlave" class="org.springframework.jms.core.JmsTemplate" parent="jmsTemplateToSlave">
    <property name="defaultDestinationName" value="defaultRequest"/>
</bean>


<!-- Response Test Start -->
<int:channel id="groupRuleReplyChannel">
    <!-- <int:queue/> -->
</int:channel>

<int-jms:inbound-channel-adapter channel="groupRuleReplyChannel" jms-template="jmsTemplateFromSlave">
    <int:poller id="defaultPoller" default="true" max-messages-per-poll="1" fixed-rate="3000"  />
</int-jms:inbound-channel-adapter>

<!-- define aggregatedReplyChannel -->
<int:channel id="aggregatedGroupRuleReplyChannel">
    <int:queue/>
</int:channel>

<int:aggregator ref="partitionHandler"
                input-channel="groupRuleReplyChannel"
                output-channel="aggregatedGroupRuleReplyChannel"
                send-timeout="3600000"/>

  2. Slave

    <int:channel id="requestsChannel" />

<bean id="connectionFactory" class="org.apache.activemq.spring.ActiveMQConnectionFactory">
    <property name="brokerURL" value="${spring.activemq.broker-url}" />
    <property name="trustAllPackages" value="true" />
</bean>

 <int-jms:message-driven-channel-adapter id="jmsIn" destination-name="#{args[0]}" channel="requestsChannel" connection-factory="connectionFactory" max-messages-per-task="1"/>

<int:service-activator input-channel="requestsChannel" output-channel="replyChannel" ref="stepExecutionRequestHandler" />

<int:channel id="replyChannel" />

<int-jms:outbound-channel-adapter connection-factory="connectionFactory" destination-name="#{args[1]}" channel="replyChannel" />

Please advice if you have experience the issue.

Let me know if you need more information.

Note: I already search a lot at here and google but no luck for solution yet.

Nghia Do
  • 2,588
  • 2
  • 17
  • 31

1 Answers1

1

ActiveMQ uses a prefetch of 1000 by default see here.

In other words, the first (up to) 1000 partitions will go to the first consumer etc.

You can reduce the prefetch; 1 is probably fine for this application.

Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • Hi Gray, could you please advice where we should apply prefetch? – Nghia Do Jun 13 '18 at 22:06
  • Did you read the documentation I pointed you to? `>To change the prefetch limit for queue consumers only configure the connection URI as follows:` `tcp://localhost:61616?jms.prefetchPolicy.queuePrefetch=1` – Gary Russell Jun 13 '18 at 22:12
  • I just want to make sure I apply correct. I have applied it but the slave work-load is not even contributed. I tried with 5 slaves and then 1 slave pick 1 message, the second one picked 4 messages and 3 remaining slaves did not nothing – Nghia Do Jun 13 '18 at 22:56
  • Hi Gary, do you have any idea? – Nghia Do Jun 14 '18 at 11:44
  • I don't; the prefetch setting should solve the problem. – Gary Russell Jun 14 '18 at 12:53
  • Thanks Gary. Does it relate to time processing of a slave? Sometime we do see the task balance between slave if time consuming of slave is more – Nghia Do Jun 14 '18 at 13:44
  • It's hard to say but, yes, if the partitions are handled quickly you might get uneven balancing. But, if that's the case, your partitions are probably too small. – Gary Russell Jun 14 '18 at 14:06
  • My partition normally has normally 3 splits and max is 10 splits. And sometime and some environments we are seeing 1 slave handles most of work. Do we have any way to make it balance even a partition done very quick? The reason I'm asking so because we always spin around 5 slaves all the time and we don't want to uneven between slaves – Nghia Do Jun 14 '18 at 14:20
  • I have no idea why you are seeing that; double check your configuration; turn on DEBUG logging to see if it gives you any clue. I don't think I can help further. – Gary Russell Jun 14 '18 at 14:45
  • Thank you so much for your help – Nghia Do Jun 14 '18 at 14:47
  • Hi Gary, I have applied this configuration in activemq.xml and I do see it is working fine. – Nghia Do Jul 11 '18 at 19:23