Spring batch - get information about files in a directory

Question

So I'm toying around with Spring Batch for the first time and trying to understand how to do things other than process a CSV file.

Attempting to read every music file in a directory for example, I have the following code but I'm not sure how to handle the Delegate part.

@Configuration
@EnableBatchProcessing
public class BatchConfiguration {

    @Autowired
    public JobBuilderFactory jobBuilderFactory;

    @Autowired
    public StepBuilderFactory stepBuilderFactory;

    @Bean
    public MusicItemProcessor processor() {
        return new MusicItemProcessor();
    }

    @Bean
    public Job readFiles() {
        return jobBuilderFactory.get("readFiles").incrementer(new RunIdIncrementer()).
                flow(step1()).end().build();
    }

    @Bean
    public Step step1() {
        return stepBuilderFactory.get("step1").<String, String>chunk(10)
                .reader(reader())
                .processor(processor()).build();
    }

    @Bean
    public ItemReader<String> reader() {
        Resource[] resources = null;
        ResourcePatternResolver patternResolver = new PathMatchingResourcePatternResolver();
        try {
            resources = patternResolver.getResources("file:/music/*.flac");
        } catch (IOException e) {
            e.printStackTrace();
        }

        MultiResourceItemReader<String> reader = new MultiResourceItemReader<>();
        reader.setResources(resources);
        reader.setDelegate(new FlatFileItemReader<>()); // ??

        return reader;
    }

}

At the moment I can see that resources has a list of music files, but looking at the stacktrace I get back, it looks to me like new FlatFileItemReader<>() is trying to read the actual content of the files (I'll want to do that at some point, just not right now).

At the moment I just want the information about the file (absolute path, size, filename etc), not what's inside.

Have I gone completely wrong with this? Or do I just need to configure something a little different?

Any examples of code that does more than process CSV lines would also be awesome

if so, FlatFileItemReader is not for you in this case, you need something like DirectorReader which is to return you a list of files (meta data) only. And then in Processor you will deal with file content ... — Nghia Do, Jun 16 '18 at 00:25
That's a good starting point, I don't know which reader I'm supposed to be using so thanks. I can't actually find any information on `DirectorReader` though? — Chris, Jun 16 '18 at 00:27
Try https://stackoverflow.com/questions/31700520/how-to-read-all-files-in-a-folder-with-spring-batch-and-multiresourceitemreader — Nghia Do, Jun 16 '18 at 15:29
That doesn't really help explain how to process a list of files... and goes back to using what you said I shouldn't be using, the `FlatFileItemReader`? — Chris, Jun 16 '18 at 21:07

score 1 · Accepted Answer · answered Jun 17 '18 at 01:23

After scouring the internet I've managed to pull together something that I think works... Some feedback would be welcome.

@Configuration
@EnableBatchProcessing
public class BatchConfiguration {

    @Autowired
    public JobBuilderFactory jobBuilderFactory;

    @Autowired
    public StepBuilderFactory stepBuilderFactory;

    @Bean
    public VideoItemProcessor processor() {
        return new VideoItemProcessor();
    }

    @Bean
    public Job readFiles() {
        return jobBuilderFactory.get("readFiles")
                .start(step())
                .build();
    }

    @Bean
    public Step step() {
        try {
            return stepBuilderFactory.get("step").<File, Video>chunk(500)
                    .reader(directoryItemReader())
                    .processor(processor())
                    .build();
        } catch (IOException e) {
            e.printStackTrace();
        }

        return null;
    }

    @Bean
    public DirectoryItemReader directoryItemReader() throws IOException {
        return new DirectoryItemReader("file:/media/media/Music/**/*.flac");
    }
}

The part that had me stuck with creating a custom reader for files. If anyone else comes across this, this is how I've done it. I'm sure there are better ways but this works for me

public class DirectoryItemReader implements ItemReader<File>, InitializingBean {
    private final String directoryPath;

    private final List<File> foundFiles = Collections.synchronizedList(new ArrayList<>());

    public DirectoryItemReader(final String directoryPath) {
        this.directoryPath = directoryPath;
    }

    @Override
    public File read() {
        if (!foundFiles.isEmpty()) {
            return foundFiles.remove(0);
        }

        synchronized (foundFiles) {
            final Iterator files = foundFiles.iterator();

            if (files.hasNext()) {
                return foundFiles.remove(0);
            }
        }

        return null;
    }

    @Override
    public void afterPropertiesSet() throws Exception {
        for (final Resource file : getFiles()) {
            this.foundFiles.add(file.getFile());
        }
    }

    private Resource[] getFiles() throws IOException {
        ResourcePatternResolver patternResolver = new PathMatchingResourcePatternResolver();
        return patternResolver.getResources(directoryPath);
    }
}

The only thing you'd need to do is implement your own processor. I've used Videos in this example, so I have a video processor

@Slf4j
public class VideoItemProcessor implements ItemProcessor<File, Video> {

    @Override
    public Video process(final File item) throws Exception {
        Video video = Video.builder()
                .filename(item.getAbsoluteFile().getName())
                .absolutePath(item.getAbsolutePath())
                .fileSize(item.getTotalSpace())
                .build();

        log.info("Created {}", video);

        return video;
    }
}

Spring batch - get information about files in a directory

1 Answers1