Using SystemCommandTasklet for split the large flat file into small files

Question

Is anyone already splitted the large flat files into small files using Spring Batch SystemCommandTasklet.

And i would like to know is it really time consuming process?

We would like to split the file with 100 million records(each record contains 15 fields).

Please any help on this would be really appreciated.

Regards

Shankar

Answer 1

I do it in my talk JSR-352, Spring Batch, and You (https://www.youtube.com/watch?v=yKs4yPs-5yU). There, I use the SystemCommandTasklet in combination with OS X's split command. You can see the example configuration in the repository for that talk here: https://github.com/mminella/jsr352-springbatch-and-you.

The specific example would be as follows:

<bean id="fileSplittingTasklet" class="org.springframework.batch.core.step.tasklet.SystemCommandTasklet" scope="step">
    <property name="command" value="split -a 5 -l 10000 #{jobParameters['inputFile']} #{jobParameters['stagingDirectory']}"/>
    <property name="timeout" value="60000"/>
    <property name="workingDirectory" value="/tmp/jsr_temp"/>
</bean>

This is the preferred approach for splitting a file. The file system/OS level tools are much faster than piping the file through the java process.