In my spring batch, i'm using Oracle DB and IBM WMQ and implemented Atomikos Transaction management and Spring Partitioning.
Requirement is, need to read 2 million records from database and after processing, post the same messages(1 message size=2kb) into WMQ and update 2 million records in database as well.
Following is my code configuration. Queue configuration: `
<bean id="atomikosJmsConnectionFactory" class="com.atomikos.jms.AtomikosConnectionFactoryBean" init-method="init" destroy-method="close">
<property name="uniqueResourceName" value="WMQ_MQSeries_XA_RMI" />
<property name="xaConnectionFactory">
<bean class="com.ibm.mq.jms.MQXAConnectionFactory">
<property name="connectionNameList" value="127.0.0.1(1414)" />
<property name="queueManager" value="LWMQ" />
<property name="channel" value="LCHNL" />
<property name="transportType">
<util:constant static-field="com.ibm.msg.client.wmq.WMQConstants.WMQ_CM_CLIENT" />
</property>
<property name="clientReconnectOptions">
<util:constant static-field="com.ibm.msg.client.wmq.WMQConstants.WMQ_CLIENT_RECONNECT" />
</property>
</bean>
</property>
<property name="ignoreSessionTransactedFlag" value="${mq.ignoreSessionTransactedFlag:false}"></property>
<property name="borrowConnectionTimeout" value="300"></property>
<property name="maxPoolSize" value="50"></property>
</bean>
<bean id="jmsTemplate" class="org.springframework.jms.core.JmsTemplate">
<property name="connectionFactory" ref="atomikosJmsConnectionFactory" />
<property name="sessionTransacted" value="true"/>
<property name="sessionAcknowledgeMode" value="0" />
<property name="receiveTimeout" value="2000" />
</bean>
`
Spring Job Configuration:
<batch:step id="masterStep">
<batch:partition step="slaveStep" partitioner="customPartitioner">
<batch:handler task-executor="taskExecutor" />
</batch:partition>
</batch:step>
<batch:listeners>
<batch:listener ref="loadJobListener" />
</batch:listeners>
</batch:job>
<batch:step id="slaveStep">
<batch:tasklet transaction-manager="atomikosTransactionManager">
<batch:chunk chunk-completion-policy="chunkCompletionPolicy" />
</batch:tasklet>
</batch:step>
<bean id="taskExecutor" class="org.springframework.core.task.SimpleAsyncTaskExecutor" >
<property name="threadNamePrefix" value="L-Thread-"></property>
<property name="concurrencyLimit" value="20"></property>
</bean>
<bean id="chunkTimeout"
class="org.springframework.batch.repeat.policy.TimeoutTerminationPolicy">
<constructor-arg value="36000" />
</bean>
<bean id="commitCount"
class="org.springframework.batch.repeat.policy.SimpleCompletionPolicy">
<property name="chunkSize" value="100" />
</bean>
<bean id="chunkCompletionPolicy"
class="org.springframework.batch.repeat.policy.CompositeCompletionPolicy">
<property name="policies">
<list>
<ref bean="commitCount" />
<ref bean="chunkTimeout" />
</list>
</property>
</bean>
Here is my linux server configuration: Model : Intel(R) Xeon(R) CPU v4 @ 2.60GHz Architecture: x86_64 No of CPU : 16 RAM : 100GB
Currently, the job takes 120 min to finish the task(including 2 million messages posting into MQ and updating into database)
my Observation: Most of the time Threads are in waiting state for MQ connection. Following is the Visual-VM screenshot of threaddump.
"L-Thread-4" #101 prio=5 os_prio=0 tid=0x000000002349e000 nid=0xcabc waiting on condition [0x00000000324cd000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000771afc540> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at com.ibm.ws.xs.xio.actor.impl.FutureImpl.await(FutureImpl.java:268)
at com.ibm.ws.xs.xio.actor.impl.FutureImpl.get(FutureImpl.java:316)
at com.ibm.ws.xsspi.xio.dispatch.MessageDispatcher.askAndGet(MessageDispatcher.java:373)
at com.ibm.ws.xs.xio.actor.impl.ActorRefImpl.askAndGet(ActorRefImpl.java:106)
at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.sendMessageInternal(XIOClientCoreMessageHandler.java:372)
at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.sendMessage(XIOClientCoreMessageHandler.java:142)
at com.ibm.ws.objectgrid.client.CommonClientCoreMessageHandler.sendGetRequest(CommonClientCoreMessageHandler.java:229)
at com.ibm.ws.objectgrid.client.RemoteTransactionCallbackImpl.mapGet(RemoteTransactionCallbackImpl.java:1953)
at com.ibm.ws.objectgrid.client.RemoteCacheLoader.get(RemoteCacheLoader.java:497)
at com.ibm.ws.objectgrid.map.BaseMap.getRaw(BaseMap.java:4846)
at com.ibm.ws.objectgrid.DiffMap.lookForKey(DiffMap.java:1391)
at com.ibm.ws.objectgrid.DiffMap.get(DiffMap.java:981)
at com.ibm.ws.objectgrid.ObjectMapImpl.get(ObjectMapImpl.java:451)
at com.ibm.ws.objectgrid.ObjectMapImpl.get(ObjectMapImpl.java:411)
at com.ibm.websphere.objectgrid.spring.ObjectGridCache.get(ObjectGridCache.java:290)
How can i fix this issue to speed up my process..?
I need to finish this job process within 1 hour.
