Spring Integration Mail: Send email after all database inserts

Question

Hello I have an integration flow that splits a file line by line, transforms each line into a POJO and then inserts that POJO into a db via JDBC outbound gateway.

I want to be able to send a single email once the process of the file has completed. I currently am sending to smtpFlow channel after my jdbcOutboundGateway, however this is sending an email after every db insert.

Here is my current flow DSL

IntegrationFlow ftpFlow() {
    return IntegrationFlows.from(
            ftpSource(), spec -> spec.poller(Pollers.fixedDelay(5, TimeUnit.SECONDS)))
            .split(splitFile())
            .transform(this::transformToIndividualScore)
            .handle(jdbcOutboundGateway(null))
            .channel("smtpFlow")
            .get();

How do I get this flow to only send one email after all files have been processed in the jdbcOutboundGateway?

Here is my splitFile() method

@Bean
FileSplitter splitFile() {
    FileSplitter fs = new FileSplitter(true, false);
    fs.setFirstLineAsHeader("IndividualScore");
    return fs;

Here is my transformToIndividualScore method

@Transformer
private IndividualScore transformToIndividualScore(String payload) {
    String[] values = payload.split(",");
    IndividualScore is = new IndividualScore();
    is.setScorecardDate(values[0]);
    is.setVnSpId(values[1]);
    is.setPrimaryCat(values[2]);
    is.setSecondaryCat(values[3]);
    is.setScore(Integer.parseInt(values[4]));
    is.setActual(values[5]);
    return is;
}

score 0 · Answer 1 · answered Jan 29 '20 at 15:00

0

Add .aggregate() after the handle to assemble the results of the splits back into a single message.

answered Jan 29 '20 at 15:00

Gary Russell

166,535
14
146
179

Another way is to use a `publishSubscribeChannel()` with a `split()` and `jdbcOutboundGateway()` as one subscriber sub-flow and `Mail.outboundAdapter()` as another one. It is really impossible to do what you are asking for all the files at once: there is just no an flag to check that all the files are done. What if a new file arrive during processing existing?.. Consider to change your logic to be based on per-file basis. Or as Gary said: with some aggregator strategy to emit email message after some number of files or timeout. – Artem Bilan Jan 29 '20 at 15:16
Oh, right; sorry, I misread the question - the aggregator will send a mail for each file instead one for each insert; you need @ArtemBilan 's suggestion for one message for all files. – Gary Russell Jan 29 '20 at 15:21
when i just drop ```.aggregate()``` after ```.handle(jdbcOutboundGateway(null))``` it doesn't complete the flow, it just hangs. – Addison Joseph Jan 29 '20 at 16:13
@GaryRussell I do want it to send it after each file, so i think your answer is correct. – Addison Joseph Jan 29 '20 at 16:14
`transformToIndividualScore` - perhaps you are dropping the sequence information headers; show your transformer; if you are returning a `Message>`, you are responsible for propagating the inbound headers to the outbound message. – Gary Russell Jan 29 '20 at 16:15
@ArtemBilan thank you for your help and sorry about the misunderstanding, I do want to send email after each file is done, not after "the process" (which could be n number of files) is completed. – Addison Joseph Jan 29 '20 at 16:16
@GaryRussell added ```transformToIndividualScore``` – Addison Joseph Jan 29 '20 at 16:21
So, `publishSubscribeChannel()` is the way to go and you don't need to worry about sequence detail for an aggregator. And that `jdbcOutboundGateway` could be just as simple as `JdbcMessageHandler` - one-way to store in DB and forget. The second subscriber just with the `Mail.outboundAdapter()` will wait until the work is done in the first subscriber. – Artem Bilan Jan 29 '20 at 16:49
BTW the `FileSplitter` doesn't populate a `sequenceSize` header in case of `iterator` option. That's why an aggregator doesn't work. Looks like we still need to come up with the `FileMarkerAggregator` solution: https://jira.spring.io/browse/INT-4116 – Artem Bilan Jan 29 '20 at 16:57
1

@ArtemBilan yes! it's making sense now to me. I am very new to integration and it's a great framework. – Addison Joseph Jan 29 '20 at 17:32
1

i will update my code and post a new message with a solution shortly. – Addison Joseph Jan 29 '20 at 17:33

Addison Joseph · Answer 2 · 2020-01-29T17:27:49.863

So a solution to my problem, (kind of).

Marking iterator to false on my FileSplitter now allows the sequencing headers.

the updated splitFile() is below

@Bean
FileSplitter splitFile() {
    FileSplitter fs = new FileSplitter(false, false);
    fs.setFirstLineAsHeader("IndividualScore");
    fs.setApplySequence(true);
    return fs;
}

My intuition is telling me that the default .aggregate() release strategy must be that the message header sequenceSize == the list of messages aggregated.

When creating the FileSplitter with iterator set to true the sequenceSize is set to 0 which will never fulfill the release strategy of the default .aggregate()

However, this makes the FileSplitter use a List to store all lines of the file in memory. The aggregator is also storing another ArrayList of lines in memory..

Is there a better solution to creating a custom aggregator that handles the END FileMarker for allowing for usage of the iterator for splitting a file?

score 0 · Accepted Answer · answered Jan 29 '20 at 19:05

with the help of @ArtemBilan

I was able to use the publishSubscribeChannel() and chain 2 subscribe() methods in sequence below is the new IntegrationFlow

 @Bean
IntegrationFlow ftpFlow() {
    return IntegrationFlows.from(
            ftpSource(), spec -> spec.poller(Pollers.fixedDelay(5, TimeUnit.SECONDS)))
            .publishSubscribeChannel(channel -> channel
                    .subscribe(
                        a -> a
                                .split(splitFile())
                                .transform(this::transformToIndividualScore)
                                .handle(jdbcMessageHandler(null)))
                    .subscribe(
                        b -> b
                                .transform(this::transformToSuccessEmail)
                                .handle(emailHandler()))
            )
            .get();

Spring Integration Mail: Send email after all database inserts

3 Answers3