0

I'm using atomikos to handle global transaction between 2 different databases. I ran before with more than 1 billion records and this problem just appeared couple days ago. I guess the reason is my table increase the size so it catch atomikos config timeout. Can anyone help me please! For more detail, this is the log that I received:

{"timestamp":"2022-11-29 18:16:09.691","level":"INFO","thread":"[BagInsertionScheduler_Worker-1] ","logger":"insert.job.BagInsertQuartzJob","msg":">>>>> START JOB: bagInsertingJob - CMJobId -28979"}
{"timestamp":"2022-11-29 18:16:09.726","level":"INFO","thread":"[BagInsertionScheduler_Worker-1] ","logger":"o.s.b.c.l.support.SimpleJobLauncher","msg":"Job: [SimpleJob: [name=bagInsertingJob]] launched with the following parameters: [{Time=1669720569691, JobCMId=-28979}]"}
{"timestamp":"2022-11-29 18:16:09.728","level":"INFO","thread":"[BagInsertionScheduler_Worker-1] ","logger":"i.j.l.BagInsertPartitionerAndJobListener","msg":">>> Just Picked a Job: -28979"}
{"timestamp":"2022-11-29 18:16:09.732","level":"INFO","thread":"[BagInsertionScheduler_Worker-1] ","logger":"o.s.batch.core.job.SimpleStepHandler","msg":"Executing step: [bagInsertMasterStep]"}
{"timestamp":"2022-11-29 18:16:09.787","level":"INFO","thread":"[BagInsertionScheduler_Worker-1] ","logger":"common.util.PartitionUtil","msg":">>>> Partition: 0- Start: 0 - End: 10"}
{"timestamp":"2022-11-29 18:16:09.787","level":"INFO","thread":"[BagInsertionScheduler_Worker-1] ","logger":"common.util.PartitionUtil","msg":">>>> Partition: 1- Start: 11 - End: 21"}
{"timestamp":"2022-11-29 18:16:09.787","level":"INFO","thread":"[BagInsertionScheduler_Worker-1] ","logger":"common.util.PartitionUtil","msg":">>>> Partition: 2- Start: 22 - End: 32"}
{"timestamp":"2022-11-29 18:16:09.788","level":"INFO","thread":"[BagInsertionScheduler_Worker-1] ","logger":"common.util.PartitionUtil","msg":">>>> Partition: 3- Start: 33 - End: 39"}
{"timestamp":"2022-11-29 18:16:19.866","level":"WARN","thread":"[Atomikos:6] ","logger":"c.a.icatch.imp.ActiveStateHandler","msg":"Transaction 10.234.73.199.tm166972056980900063 has timed out and will rollback."}
{"timestamp":"2022-11-29 18:16:19.866","level":"WARN","thread":"[Atomikos:7] ","logger":"c.a.icatch.imp.ActiveStateHandler","msg":"Transaction 10.234.73.199.tm166972056981000065 has timed out and will rollback."}
{"timestamp":"2022-11-29 18:16:19.866","level":"WARN","thread":"[Atomikos:5] ","logger":"c.a.icatch.imp.ActiveStateHandler","msg":"Transaction 10.234.73.199.tm166972056980900064 has timed out and will rollback."}
{"timestamp":"2022-11-29 18:16:19.866","level":"WARN","thread":"[Atomikos:4] ","logger":"c.a.icatch.imp.ActiveStateHandler","msg":"Transaction 10.234.73.199.tm166972056980900062 has timed out and will rollback."}
{"timestamp":"2022-11-29 18:17:11.209","level":"ERROR","thread":"[SimpleAsyncTaskExecutor-43] ","logger":"o.s.batch.core.step.AbstractStep","msg":"Encountered an error executing step bagInsertingSlaveStep in job bagInsertingJob"}
org.springframework.transaction.UnexpectedRollbackException: JTA transaction unexpectedly rolled back (maybe due to a timeout); nested exception is javax.transaction.RollbackException: One or more resources refused to commit (possibly because of a timeout in the resource - see the log for details). This transaction has been rolled back instead.
    at org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1038)
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:743)
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:711)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:654)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:407)
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708)
    at insert.job.step.BagInsertUpdater$$EnhancerBySpringCGLIB$$d6e05d87.write(<generated>)
    at insert.job.step.BagInsertUpdater$$FastClassBySpringCGLIB$$e835cd44.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:137)
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708)
    at insert.job.step.BagInsertUpdater$$EnhancerBySpringCGLIB$$efdca957.write(<generated>)
    at org.springframework.batch.core.step.item.SimpleChunkProcessor.writeItems(SimpleChunkProcessor.java:193)
    at org.springframework.batch.core.step.item.SimpleChunkProcessor.doWrite(SimpleChunkProcessor.java:159)
    at org.springframework.batch.core.step.item.SimpleChunkProcessor.write(SimpleChunkProcessor.java:294)
    at org.springframework.batch.core.step.item.SimpleChunkProcessor.process(SimpleChunkProcessor.java:217)
    at org.springframework.batch.core.step.item.ChunkOrientedTasklet.execute(ChunkOrientedTasklet.java:77)
    at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:407)
    at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:331)
    at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140)
    at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:273)
    at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:82)
    at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:375)
    at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:215)
    at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:145)
    at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:258)
    at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:208)
    at org.springframework.batch.core.partition.support.TaskExecutorPartitionHandler$1.call(TaskExecutorPartitionHandler.java:138)
    at org.springframework.batch.core.partition.support.TaskExecutorPartitionHandler$1.call(TaskExecutorPartitionHandler.java:135)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.lang.Thread.run(Thread.java:831)
Caused by: javax.transaction.RollbackException: One or more resources refused to commit (possibly because of a timeout in the resource - see the log for details). This transaction has been rolled back instead.
    at com.atomikos.icatch.jta.TransactionImp.rethrowAsJtaRollbackException(TransactionImp.java:48)
    at com.atomikos.icatch.jta.TransactionImp.commit(TransactionImp.java:188)
    at com.atomikos.icatch.jta.TransactionManagerImp.commit(TransactionManagerImp.java:414)
    at com.atomikos.icatch.jta.UserTransactionImp.commit(UserTransactionImp.java:86)
    at org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1035)
    ... 39 common frames omitted
Caused by: com.atomikos.icatch.RollbackException: One or more resources refused to commit (possibly because of a timeout in the resource - see the log for details). This transaction has been rolled back instead.
    at com.atomikos.icatch.imp.ActiveStateHandler.prepare(ActiveStateHandler.java:202)
    at com.atomikos.icatch.imp.CoordinatorImp.prepare(CoordinatorImp.java:523)
    at com.atomikos.icatch.imp.CoordinatorImp.terminate(CoordinatorImp.java:687)
    at com.atomikos.icatch.imp.CompositeTransactionImp.commit(CompositeTransactionImp.java:282)
    at com.atomikos.icatch.jta.TransactionImp.commit(TransactionImp.java:172)
    ... 42 common frames omitted
{"timestamp":"2022-11-29 18:17:11.212","level":"INFO","thread":"[SimpleAsyncTaskExecutor-43] ","logger":"o.s.batch.core.step.AbstractStep","msg":"Step: [bagInsertingSlaveStep:partition3] executed in 1m1s410ms"}

I try to expand the config timeout from 1000 to 10000, but it does not work. Hope that anyone can explain what's happened and how can i fix it.

leetrinh
  • 23
  • 2

1 Answers1

0

You can configure your JTA settings to increase the timeouts by tweaking your transactions.properties (can be placed in your main resources folder) The configurations of interest are:

Property name Description Since
com.atomikos.icatch.max_timeout Specifies the maximum timeout (in milliseconds) that can be allowed for transactions. Defaults to 300000. This means that calls to UserTransaction.setTransactionTimeout() with a value higher than configured here will be max'ed to this value. For 4.x or higher, a value of 0 means no maximum (i.e., unlimited timeouts are allowed). NOTE: as of 5.0, using 0 interferes with recovery. Instead, use Long.MAX_VALUE to specify unlimited. 3.x, 4.x
com.atomikos.icatch.default_jta_timeout The default timeout for JTA transactions (optional, defaults to 10000 ms) 3.4, 4.x

So if you configured your transactions.properties file with:

# transactions.properties
# .../src/main/resources/transactions.properties
# Max timeout for any JTA transactions. (10 minutes)
com.atomikos.icatch.max_timeout=600000
# Default timeout for any JTA transactions. (5 minutes)
com.atomikos.icatch.default_jta_timeout=300000

See: Atomikos - JtaProperties - Transaction manager

u-ways
  • 6,136
  • 5
  • 31
  • 47