How to process files of different formats using spring batch?

Question

My requirement is to poll two different directories for different format files (PSV and CSV) and to process it using spring batch.

I used inbound channel adapter to poll the directory . But , i couldnt able to find a way to call corresponding reader and its tokenizer and field mapper based on the type of file .

e.g. if it is psv - call PSV reader , PSV line mapper if it is csv - call CSVReader , CSV line mapper

Any help would be appreciated .

Thanks

the solution is to create your own reader and apply the corresponding logic based on the file extension — Rafik BELDI, Nov 20 '15 at 10:34
Check http://stackoverflow.com/questions/33777596/parsing-several-csv-files — Luca Basso Ricci, Nov 20 '15 at 12:59

Rafik BELDI · Answer 1 · 2015-11-20T15:19:04.410

You can create your Own Reader : CustomFilesReader :

public class CustomFilesReader implements InitializingBean{

    private List<File> yourFiles= null;

    public File read() {
        if ((yourFiles!= null) && (yourFiles.size() != 0)) {  
          return yourFiles.remove(0);
        }
        return null;
    }

    //Reading Items from your dir
    private void loadItems() {
        this.yourFiles= new ArrayList<File>();
       // populate the items
     }


    @Override
    public void afterPropertiesSet() throws Exception {
        loadItems();
    }
}

Register the bean :

<bean id="customFilesReader " class="mypackage.CustomFilesReader "/>


<bean id="myReader"
    class="org.springframework.batch.item.adapter.ItemReaderAdapter">
    <property name="targetObject" ref="customFilesReader " />
    <property name="targetMethod" value="read" />
</bean>

The Same thing can be Done for a custom ItemProcessor and a custom ItemWriter.

score 1 · Accepted Answer · answered Nov 21 '15 at 14:02

Thank you for all your suggestions.

I followed a different approach. I have setted the item reader and writer as a job parameter. So, based on file type , respective item reader and writer will be invoked

<chunk reader= "#{JobParameter[bean.reader]}" writer="#{JobParameter[bean.writer]}"/>

Thanks

score 0 · Answer 3 · answered Nov 20 '15 at 12:01

An alternative would be to create a class similar to ClassifierCompositeItemProcessor or ClassifierCompositeItemWriter. That is to say, it would take a Classifier, a list of delegated ItemReader and choose the right ItemReader depending of the result of the Classifier.

Problem is, you don't have an input to classify, since the read isn't done yet. That's why I thought about using the resource property as the element to classify. Here's the result :

public class ResourceClassifierCompositeItemReader<T> implements ItemReader<T> {

    private Classifier<String, ItemReader<? extends T>> classifier = new ClassifierSupport<String, ItemReader<? extends T>> (null);
    private Resource resource;

    public void setClassifier(Classifier<String, ItemReader<? extends T>> classifier) {
        this.classifier = classifier;
    }

    @Override
    public T read() throws Exception, UnexpectedInputException, ParseException {
        return readItem(classifier.classify(resource.getFilename()));
    }

    private T readItem(ItemReader<? extends T> reader) throws Exception {
        return reader.read();
    }

    public void setResource(Resource resource) {
        this.resource = resource;
    }

}

Now for how to use it. First you'll need a MultiResourceItemReader to read both .PSV and .CSV at the same time. Then you'll have to delegate the read to the ResourceClassifierCompositeItemReader. You'll have to add a BackToBackPatternClassifier to classify the String resource.getFilename() (i.e. the file name) and call an ItemReader accordingly (via a MatcherMap). To do that, you'll need to write your own RouterDelegate (taking the filename, split it to get the extension, and return it as a String to be matched).

How to process files of different formats using spring batch?

3 Answers3