Spring batch jsr 352 how to prevent partitioned job from leaving thread alive which prevent process from ending

Question

Let me explain how my app is set up. First I have a stand alone command line started app that runs a main which in turn calls start on a job operator passing the appropriate params. I understand the start is an async call and once I call start unless I block some how in my main it dies.

My problem I have run into is when I run a partitioned job it appears to leave a few threads alive which prevents the entire processing from ending. When I run a non-partitioned job the process ends normally once the job has completed.

Is this normal and/or expected behavior? Is there a way to tell the partitioned threads to die. It seems that the partitioned threads are blocked waiting on something once the job has completed and they should not be?

I know that I could monitor for batch status in the main and possibly end it but as I stated in another question this adds a ton of chatter to the db and is not ideal.

An example of my job spec

<job id="partitionTest" xmlns="http://xmlns.jcp.org/xml/ns/javaee" version="1.0">
<step id="onlyStep">
    <partition>
        <plan partitions="2">
            <properties partition="0">
                <property name="partitionNumber" value="1"></property>
            </properties>
            <properties partition="1">
                <property name="partitionNumber" value="2"></property>
            </properties>
        </plan>
    </partition>

    <chunk item-count="2">
        <reader id="reader" ref="DelimitedFlatFileReader">
            <properties>
                <!-- Reads in from file Test.csv -->
                <property name="fileNameAndPath" value="#{jobParameters['inputPath']}/CSVInput#{partitionPlan['partitionNumber']}.csv" />
                <property name="fieldNames" value="firstName, lastName, city" />
                <property name="fullyQualifiedTargetClass" value="com.test.transactionaltest.Member" />
            </properties>
        </reader>
        <processor ref="com.test.partitiontest.Processor" />
        <writer ref="FlatFileWriter" >
            <properties>
                <property name="appendOn" value="true"/>
                <property name="fileNameAndPath" value="#{jobParameters['outputPath']}/PartitionOutput.txt" />
                <property name="fullyQualifiedTargetClass" value="com.test.transactionaltest.Member" />
            </properties>
        </writer>
    </chunk>
</step>
</job>

Edit:

Ok reading a bit more about this issue and looking into the spring batch code, it appears there is a bug at least in my opinion in the JsrPartitionHandler. Specifically the handle method creates a ThreadPoolTaskExecutor locally but then that thread pool is never cleaned up properly. A shutdown/destroy should be called before that method returns in order to perform some clean up otherwise the threads get left in memory and out of scope.

Please correct me if I am wrong here but that definitely seems like what the problem is.

I am going and try to make a change regarding it and see how it plays out. I'll update after I have done some testing.

Mark Kouba · Accepted Answer · 2015-09-30T16:28:43.723

I have confirmed this issue to be a bug (still in my opinion atm) in the spring batch core lib.

I have created a ticket over at the spring batch jira site. There is a simple attached java project to the ticket that confirms the issue I am seeing. If any one else runs into the problem they should refer to that ticket.

I have found a temporary work around that just uses a wait/notify scheme and it seems once added that the pooled threads shut down. I'll add each of the classes/code and try and explain what I did.

In main thread/class, this was code that lived in the main method or a method called from main

        while(!ThreadNotifier.instance(this).getNotify()){
        try {
            synchronized(this){
                System.out.println("WAIT THREAD IS =======" + Thread.currentThread().getName());
                wait();
            }
        } catch (InterruptedException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
    }

This is the ThreadNotifier class

public class ThreadNotifier {


private static ThreadNotifier tn = null;

private boolean notification = false;

private Object o;

private ThreadNotifier(Object o){
    this.o = o;

}

public static ThreadNotifier instance(Object o){

    if(tn == null){
        tn = new ThreadNotifier(o);
    }

    return tn;
}


public void setNotify(boolean value){
    notification = true;
    synchronized(o){
        System.out.println("NOTIFY THREAD IS =======" + Thread.currentThread().getName());
        o.notify();
    }
}

public boolean getNotify(){
    return notification;
}

}

And lastly this is a job listener that I used to provide the notification back

public class PartitionWorkAround implements JobListener {

@Override
public void beforeJob() throws Exception {
    // TODO Auto-generated method stub

}

@Override
public void afterJob() throws Exception {

    ThreadNotifier.instance(null).setNotify(true);

}

}

This best I could come up with until the issue is fixed. For reference I used knowledge learned about guarded blocks here to figure out a way to do this.

Spring batch jsr 352 how to prevent partitioned job from leaving thread alive which prevent process from ending

1 Answers1