Limit number of threads using split in Spring Batch

Question

I have the following spring batch xml file, in which I define a job in which I want to run 4 different tasks in parallel:

    <split id="fillNDDependencies" task-executor="asyncTaskExecutor"
        next="decisionExecuteNDDependencies">
        <flow>
            <step id="fillTABLE1">
                <tasklet ref="runTABLE1Tasklet" />
            </step>
        </flow>
        <flow>
            <step id="fillTABLE2">
                <tasklet ref="runTABLE2Tasklet" />
            </step>
        </flow>
        <flow>
            <step id="fillTABLE3">
                <tasklet ref="runTABLE3Tasklet" />
            </step>
        </flow>
        <flow>
            <step id="fillTABLE4">
                <tasklet ref="runTABLE4Tasklet" />
            </step>
        </flow>
    </split>

I use a ThreadPoolTaskExecutor to control the number of threads used.

<bean id="asyncTaskExecutor"
    class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor">
    <property name="corePoolSize" value="1"></property>
    <property name="maxPoolSize" value="1"></property>
    <property name="queueCapacity" value="0"></property>
    <property name="keepAliveSeconds" value="10"></property>
</bean>

Here is the issue: ideally I want to run four tasks in parallel, but there will be times in which my machine will be more "busy", and I will want to execute just two of these tasks at the same time, depending on the number of threads I have available in my machine, but without changing the current split condition.

I tried to limit the corePoolSize and maxPoolSize in my task executor, but I get the following error when I execute the job:

org.springframework.batch.core.JobExecutionException: Flow execution ended unexpectedly
    at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:140)
    at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:304)
    at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:135)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.springframework.batch.core.job.flow.FlowExecutionException: TaskExecutor rejected task for flow=fillNDDependencies1.1
    at org.springframework.batch.core.job.flow.support.state.SplitState.handle(SplitState.java:103)
    at org.springframework.batch.core.configuration.xml.SimpleFlowFactoryBean$DelegateState.handle(SimpleFlowFactoryBean.java:207)
    at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:165)
    at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:144)
    at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:134)
    ... 5 more

So, unless the PoolSize matches the number of steps inside the split, the job does not work.

How am I able to limit the tasks executed at a time, without having to create a new job with less steps inside each split?

Thank you.

Note that this is not a duplicate of this issue How to set up multi-threading in Spring Batch? , as I am already using split as a means to parallelize my task executions.

https://docs.spring.io/spring-batch/trunk/reference/html/scalability.html#multithreadedStep may help. — Raj, Oct 01 '18 at 09:33
@Raj , I have added a line at the end to show how this question defers from the one you suggested. — manuel mourato, Oct 01 '18 at 09:40

score 0 · Answer 1 · answered Oct 01 '18 at 10:47

The ThreadPoolTaskExecutor has a parameter called queueCapacity which is used to set how many tasks can the executor accept before starting to reject tasks. You need to set this parameter according to your needs.

Now for the number of worker threads, I see in your example you set the corePoolSize as well as the maxPoolSize to 1, so there will be at most one thread in the pool. You can set the corePoolSize to 1 and the maxPoolSize to 2 for example if you want two threads to work in parallel.

These settings can be changed dynamically at runtime through JMX. See Javadoc here: https://docs.spring.io/spring/docs/current/javadoc-api/org/springframework/scheduling/concurrent/ThreadPoolTaskExecutor.html#setMaxPoolSize-int-

Hope this helps.

Limit number of threads using split in Spring Batch

1 Answers1