File data validation before moving to the target system

Question

I want to validate the data(to maintain data integrity) inside the file using some logic before sending to target system. For that I am using files and its checksum files logic like this, abc.txt is the original file abc.txt.checksum is the checksum file of it. The abc.txt.checksum is the hash of abc.txt file and will be used to validate before sending to the target system.Both files will be reside in the same source location. I have a filter condition for not to pick .checksum file while poling as I dont want to sent .checksum file to the target system. Once abc.txt file is picked, i am using one java logic to create hash of this file internally and store it into the in-memory and it has to be compared with the checksum(abc.txt.checksum) data which is already there in that location. If both checksum data matches , then only the original file has to be moved to target system else the it should not process and throws error. Could someone please guide me how to achieve this? or How will i get the .checksum files (abc.txt.checksum) as it will be there in the source location always?

Answer 1

What you explain is fully as a logic for the aggregator component.

You poll for all the files in the directory, group them by the file name prefix. It can be done via a SpEL expression for the correlationKey against a FileHeaders.FILENAME message header. This way the abc.txt and abc.txt.checksum will be grouped together. Then you can produce an output message from the aggregator via MessageGroupProcessor the way that checksum is stored in the header and with original file as a payload of a single message to produce.

Afterwards you are good to use your validation logic and there is no reason to worry about one more directory round trip.

The point of an aggregator that any file is not going to be emitted downstream until its pair arrives.

See more info in the Reference Manual: https://docs.spring.io/spring-integration/reference/html/#aggregator

UPDATE

Since you talk about two files (original and checksum for it), then you have a very simple ReleaseStrategy - (group) -> group.size() == 2. The CorrelationStrategy can be based on the FileHeaders.FILENAME header which is present together with the File payload after FileReadingMessageSource producing. That function logic might be like this:

(message) -> message.getHeaders().get(FileHeaders.FILENAME, String.class).replaceFirst("\.checksum", "")