0

I am using Spring Batch (version 4.1.2). The problem is the following:

1/ batch job is step oriented (2 steps), the first step loads data from oracle database (customer orders - the only small amount of data), filters them for further processing and stores the result into an in-memory object (orderNo - i.e. process this order in the second step)

2/ second step takes these filtered data from memory and processes them - loads data (now full amount of data, full order) and process them - starts stored procedure in oracle and then calls some external web service

The first step is very fast - approx. 5 sec.

The second step is very slow - for every customer order, it takes approx. 10 sec.

Typical size of data: the First step loads thousands of orders and filters approx. 100-300 orders for further processing in the second step.

Configuration for the first step: a chunk of 100 orders

Configuration for second step: chunk = 1

And the problem: If the second step (calling web service) failes, the metadata database transaction fails because of timeout. Or if the job completes OK, again the metadata database fails with timeout. Why? The transaction timeout is set to 30 seconds. It looks to me that the transaction of the metadata database is not affected by the chunk configuration.

Note: order database: oracle, metadata database: postgresql

Any suggestions, solutions, etc, please?

mirek stohr
  • 1
  • 2
  • You said "second step takes these filtered data from memory". What does it exactly mean? Are you saving it to spring jobExecutionContext or something? – qristjan Nov 06 '19 at 12:00
  • The first step decides if customer order should be processed or not. If yes, its orderNo (long) is stored in spring bean, its instance is shared. The second step takes these orderNos from this bean and process them - one after another. – mirek stohr Nov 06 '19 at 12:35
  • Maybe important: The spring batch job runs on jboss (version JBoss EAP 7.1.5.GA) – mirek stohr Nov 06 '19 at 13:43

1 Answers1

0

@Transactional(propagation = Propagation.REQUIRES_NEW)

on ItemProcessor method

O process(I var1) throws Exception;

was solution of the problem.

mirek stohr
  • 1
  • 2