Reading multiple files resides in a file system which matches the job parameters using MultiResourceItemReader

Question

Use Case : I would like to launch a job which takes employee id as job parameters, which will be multiple employee ids. In a file system, files will be residing which contains employee ids as part of the file name (It is a remote file system, not local) i need to process those files where file name contains the employee-id and passing it to the reader.

I am thinking of using MultiResourceItemReader but i am confused how to match the file name with Employee Id (Job Parameter) which is there in a file system.

Please suggest.

to read all files in a folder with spring-batch, visit: https://stackoverflow.com/questions/31700520/how-to-read-all-files-in-a-folder-with-spring-batch-and-multiresourceitemreader/31710152 — zedtimi, Jun 05 '18 at 14:42

Thrax · Answer 1 · 2016-01-13T15:27:30.567

The class MultiResourceItemReader has a method setResources(Resources[] resources) which lets you specify resources to read either with an explicit list or with a wildcard expression (or both).

Example (explicit list) :

<bean class="org.springframework.batch.item.file.MultiResourceItemReader">
    <property name="resources">
        <list>
             <value>file:C:/myFiles/employee-1.csv</value>
             <value>file:C:/myFiles/employee-2.csv</value>
        </list>
    </property>
</bean>

Example (wildcard) :

<bean class="org.springframework.batch.item.file.MultiResourceItemReader">
    <property name="resources" value="file:C:/myFiles/employee-*.csv" />
</bean>

As you may know, you can use job parameters in configuration by using #{jobParameters['key']} :

<bean class="org.springframework.batch.item.file.MultiResourceItemReader">
    <property name="resources" value="file:C:/myFiles/employee-#{jobParameters['id']}.csv" />
</bean>

Unfortunately, wildcard expressions can't manage an OR expression over a list of value with a separator (id1, id2, id3...). And I'm guessing you don't know how many distinct values you'll have to declare an explicit list with a predefined number of variables.

However a working solution would be to use the Loop mechanism of Spring Batch with a classic FlatFileItemReader. The principle is basically to set the next="" on the last step to the first step until you have exhausted every item to read. I will provide code samples if needed.

EDIT

Let's say you have a single chunk to read one file at a time. First of all, you'd need to put the current id from the job parameter in the context to pass it to the reader.

public class ParametersManagerTasklet implements Tasklet, StepExecutionListener {

    private Integer count = 0;
    private Boolean repeat = true;

    @Override
    public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {

        // Get job parameter and split
        String[] ids = chunkContext.getStepContext().getJobParameters().getString(PARAMETER_KEY).split(DELIMITER); 
        // Check for end of list
        if (count >= ids.length) {
            // Stop loop
            repeat = false;
        } else {
            // Save current id and increment counter
            chunkContext.getStepContext().getJobExecutionContext().put(CURRENT_ID_KEY, ids[count++]; 
        }
    }

    @Override
    public ExitStatus afterStep(StepExecution stepExecution) {

        if (!repeat) {
            return new ExitStatus("FINISHED");
        } else {
            return new ExitStatus("CONTINUE");
        }
    }
}

Now you declare this step in your XML and create a loop :

<batch:step id="ParametersStep">
    <batch:tasklet>
        <bean class="xx.xx.xx.ParametersManagerTasklet" />
    </batch:tasklet>
    <batch:next on="CONTINUE" to="ReadStep" />
    <batch:end on="FINISHED" />
</batch:step>

<batch:step id="ReadStep">
    <batch:tasklet>
        <batch:chunk commit-interval="10">
            <batch:reader>
                <bean class="org.springframework.batch.item.file.MultiResourceItemReader">
                    <property name="resources" value="file:C:/myFiles/employee-#{jobExecutionContext[CURRENT_ID_KEY]}.csv" />
                </bean>
            </batch:reader>
            <batch:writer>
            </batch:writer>
        </batch:chunk>
    </batch:tasklet>
    <batch:next on="*" to="ParametersStep" />
</batch:step>

Thanks a ton for the reply and nice explanation. I have tried with wildcard expression. As u pointed out it works with either a single jobparameter or with *.xml or *.csv. It would be very helpful if you provide some code samples. — Jay, Jan 13 '16 at 13:59
You can try with a `FactoryBean` (https://spring.io/blog/2011/08/09/what-s-a-factorybean) for a `Resource[]` starting from job-parameters. But is just an idea because I haven't tried — Luca Basso Ricci, Jan 13 '16 at 14:11
@Thrax Works exactly as expected... But as it takes a extra tasklet thinking to go with FactoryBean... +1 for the answer though... — Jay, Jan 19 '16 at 05:59

Luca Basso Ricci · Answer 2 · 2016-01-14T08:54:32.633

You can write your own FactoryBean to perform a custom resources search.

public class ResourcesFactoryBean extends AbstractFactoryBean<Resource[]> {
    String[] ids;
    String path;

    public void setIds(String[] ids) {
        this.ids = ids;
    }

    public void setPath(String path) {
        this.path = path;
    }

    @Override
    protected Resource[] createInstance() throws Exception {
        final List<Resource> l = new ArrayList<Resource>();
        final PathMatchingResourcePatternResolver x = new PathMatchingResourcePatternResolver();
        for(final String id : ids)
        {
            final String p = String.format(path, id);
            l.addAll(Arrays.asList(x.getResources(p)));
        }
        return l.toArray(new Resource[l.size()]);
    }

    @Override
    public Class<?> getObjectType() {
        return Resource[].class;
    }
}

---

<bean id="reader" class="org.springframework.batch.item.file.MultiResourceItemReader" scope="step">
  <property name="delegate" ref="itemReader" />
    <property name="resources">
      <bean class="ResourcesFactoryBean">
        <property name="path"><value>file:C:/myFiles/employee-%s.cvs</value>    </property>
        <property name="ids">
          <value>#{jobParameters['id']}</value>
        </property>
      </bean>
   </property>
</bean>

jobParameter 'id' is a comma separated list of your ID.

Perfectly fits for my problem... Thanks a lot.... +1 for simple solution(If possible want to +100) — Jay, Jan 19 '16 at 06:01
Awesome, thanks for the example! Looks much simpler than doing a loop. +1 — Thrax, Jan 19 '16 at 07:57

Reading multiple files resides in a file system which matches the job parameters using MultiResourceItemReader

2 Answers2