0

I'm reading files line by line using FileSplitter, and then filter some of the corrupted lines - as to discard some lines - in order to do webservices calls for each line.

What I need to do in addition to this is to get the file moved to completed files directory.

Update: Just to be more clear, the successful/good lines after filtering will need to be aggregated in a file to be written to the completed directory and the corrupted/filtered lines will be written to another one.

Mentioning the above, I was not clear about the following points:

  • Will I need some kind of correlation between the messages/lines as to provide it manually or Spring Integration will do it for me?
  • If some lines are discarded due to the filtering, what is the right strategy to use with the aggregator?
  • How can I make use of file markers as to identify when the file is completed to move to the completed directory?

The scenario simply looks like:

FileSplitter -> line Filter -> Outbound Gateway (webservices) -> Aggregator
Ahmed Ali
  • 105
  • 1
  • 8

1 Answers1

0

The FileSplitter can be configured to emit FileSplitter.FileMarker as the first message with the FileSplitter.FileMarker.Mark.START and in the end with the FileSplitter.FileMarker.Mark.END respectively.

You can add a router (PayloadTypeRouter should be enough) after splitter to handle FileSplitter.FileMarker in a different flow. And if it is a FileSplitter.FileMarker.Mark.END you just perform a desired logic to move to the completed directory.

I don't think that you need an aggregator for this scenario.

Anyway the FileSplitter can be configured with the setApplySequence(true) to be able to correlate a group downstream in the aggregator. But you still won't be able to know the size of the group to release it in time. The FileSplitter.FileMarker.Mark.END still can help here, although I don't see the reason in that for your task.

There is some info on the matter in the Reference Manual.

Artem Bilan
  • 113,505
  • 11
  • 91
  • 118
  • 1
    See the [`file-split-ftp` sample app](https://github.com/spring-projects/spring-integration-samples/tree/master/applications/file-split-ftp) for an example of using markers to determine the end of processing (in that case it sends 3 parts of the file to different FTP servers). – Gary Russell Apr 11 '18 at 16:26
  • Thanks guys for your answers, but i still have a concern regarding having the FileMarkers routed to the aggregator through a different flow than the other lines, because of the delays in the Outbound Gateway (webservices) requests/responses. will the 'END' FileMarker arrive to the aggregator before the last line coming back from the Outbound Gateway? – Ahmed Ali Apr 19 '18 at 15:14
  • That's fine too. The `FileMarker.END` will have an information how much line it has read. So, you will need to implement a smart `ReleaseStrategy` to check for the `FileMarker.END` presence in the group and compare that `lineCount` with the size of the group. Therefore even if `FileMarker.END` will arrive to the aggregator much earlier then other processed lines, you still won't come out from the aggregator because you are waiting for all the lines. – Artem Bilan Apr 19 '18 at 15:23
  • I think what you need here should be implemented with the https://jira.spring.io/browse/INT-4116 – Artem Bilan Apr 19 '18 at 15:27