3

I've been trying to add a timeout in the gather to don't wait that every flow finished. but when I added the timeout doesn't work because the aggregator waits that each flow finished.

@Bean
public IntegrationFlow queueFlow(LogicService service) {    
        return f -> f.scatterGather(scatterer -> scatterer
                                .applySequence(true)
                                .recipientFlow(aFlow(service))
                                .recipientFlow(bFlow(service))
                        , aggregatorSpec -> aggregatorSpec.groupTimeout(2000L)) 

E.g of my flows one of them has 2 secs of delay and the other one 4 secs

public IntegrationFlow bFlow(LogicService service) {
        return IntegrationFlows.from(MessageChannels.executor(Executors.newCachedThreadPool()))
                .handle(service::callFakeServiceTimeout2)
                .transform((MessageDomain.class), message -> {
                    message.setMessage(message.getMessage().toUpperCase());
                    return message;
                }).get();
    } 

I use Executors.newCachedThreadPool() to run parallel. I'd like to release each message that was contained until the timeout is fulfilled

Another approach that I've been testing was to use a default gatherer and in scatterGather set the gatherTimeout but I don't know if I'm missing something Approach gatherTimeout

UPDATE

All the approaches given in the comments were tested and work normally, the only problem is that each action is evaluated over the message group creation. and the message group is created just until the first message arrived. The ideal approach is having an option of valid at the moment when the scatterer distributes the request message.

My temporal solution was to use a release strategy ad hoc applying a GroupConditionProvider which reads a custom header that I created when I send the message through the gateway. The only concern of this is that the release strategy only will be executed when arriving at a new message or I set a group time out.

1 Answers1

0

The groupTimeout on the aggregator is not enough to release the group. If you don't get the whole group on that timeout, then it is going to be discarded. See sendPartialResultOnExpiry option: https://docs.spring.io/spring-integration/reference/html/message-routing.html#agg-and-group-to

If send-partial-result-on-expiry is true, existing messages in the (partial) MessageGroup are released as a normal aggregator reply message to the output-channel. Otherwise, it is discarded.

The gatherTimeout is good to have if you expect no replies from the gatherer at all. So, this way you won't block the scatter-gather thread forever: https://docs.spring.io/spring-integration/reference/html/message-routing.html#scatter-gather-error-handling

Artem Bilan
  • 113,505
  • 11
  • 91
  • 118
  • But when I put that option I don't receive any message from the gather. I'd like to release all messages that are contained in the gather until the timeout is over – jandresboyaca Oct 06 '21 at 20:12
  • Are you sure that your service answers during those 2 seconds? Amy you need to wait a little more? – Artem Bilan Oct 06 '21 at 20:13
  • this is the code https://github.com/jandresboyaca/SpringIntegration/blob/master/src/main/java/com/deploysoft/cloud/flows/QueueIntegration.java – jandresboyaca Oct 06 '21 at 20:17
  • My goal is to release the messages that are in that moment and discard the others. So I have two flows one that takes 2 seconds and one that takes 4. My timeout could be 3 sec so I try to return one message of the flow of 2 sec and discard the message of the flow that takes 4 sec – jandresboyaca Oct 06 '21 at 20:19
  • But I don't see `groupTimeout()` in your configuration. The `gatherTimeout()` does not help here: an aggregator still waits for all the messages to collect. And again: it is not perfect to test 2 seconds delay and wait for 2 seconds. The delay in reality might be a bit longer since resources busyness on your CPU. – Artem Bilan Oct 06 '21 at 20:22
  • Better to assume that service is going to answer within 1 second if you still want to wait for 2 seconds. – Artem Bilan Oct 06 '21 at 20:23
  • I don't chat. I have work to do and answer here whenever I have time. Sorry – Artem Bilan Oct 06 '21 at 20:24
  • Sry my bad. Now returns the first message and discard the other one and takes more time, I mean groupTimeout is the time to wait until the other message, Does the aggregator always wait for all messages? Isn't there a way to free the group depending on a global timeout? Or would you have to create an ad hoc release strategy? – jandresboyaca Oct 06 '21 at 20:29
  • Oh! Yes; group timeout is re-started after each received message. Maybe you don't need to wait for the second reply at all and then simple `MessageCountReleaseStrategy` is enough for you? – Artem Bilan Oct 06 '21 at 20:33
  • And what do you think of `TimeoutCountSequenceSizeReleaseStrategy`? – jandresboyaca Oct 06 '21 at 20:56
  • That's your choice. See its JavaDocs. Only the problem that `ReleaseStrategy` is not triggered until the message arrives to the aggregator. That was one of the reasons why we have introduced a `groupTimeout` feature. – Artem Bilan Oct 06 '21 at 20:59
  • That is a concern because I would have to wait N additional times for each message to validate the release strategy. Long story short there is no global time out that indicates that at the N time release the entire group. – jandresboyaca Oct 07 '21 at 00:59
  • There is. See `MessageGroupStoreReaper`: https://docs.spring.io/spring-integration/docs/current/reference/html/message-routing.html#reaper – Artem Bilan Oct 07 '21 at 01:50
  • In DSL , reaper will be all the configurations that I sent in messageStore method? by chance, do you have an example in DSL? – jandresboyaca Oct 07 '21 at 02:25
  • Not sure what the question. You declare `MessageGroupStoreReaper` `@Bean`. Inject into it the same `MessageGroupStore` you inject into the `gatherer`. And also see `@Scheduled` from Spring how to have a task in Java configuration : https://docs.spring.io/spring-framework/docs/current/reference/html/integration.html#scheduling-annotation-support-scheduled – Artem Bilan Oct 07 '21 at 02:31
  • Artem Thanks for your help This was my solution, Any suggestions? https://github.com/jandresboyaca/SpringIntegration/blob/master/src/main/java/com/deploysoft/cloud/flows/QueueIntegration.java#L56 – jandresboyaca Oct 07 '21 at 21:23
  • Looks good! But you don't need that `.taskScheduler(taskSchedulerTest)` for `gatherer` since you don't deal with `groupTimeout` any more! – Artem Bilan Oct 07 '21 at 21:29
  • Arthem I tested my approach but I don't know if I'm wrong Because the repeat doesn't work as I expect . I explained my approach in the code https://github.com/jandresboyaca/SpringIntegration/blob/master/src/main/java/com/deploysoft/cloud/flows/QueueIntegration.java#L56 – jandresboyaca Oct 08 '21 at 01:13
  • I see. Try to `expireGroupsUponTimeout` to `false` on the `gather`. See its explanation in the docs: https://docs.spring.io/spring-integration/docs/current/reference/html/message-routing.html#aggregator-xml - #23: `When a group is completed due to a timeout (or by a MessageGroupStoreReaper), the group is expired (completely removed) by default. Late arriving messages start a new group.`. That's probably what you are experiencing. – Artem Bilan Oct 08 '21 at 16:58
  • I think it is due to something in the message group, what I noticed was that the message group is created as soon as it receives the first message, so it is not taking into account the time of the first flow. I add expireGroupsUponTimeout but I have the same behavior – jandresboyaca Oct 08 '21 at 18:15
  • I think I will have to use an ad hoc release strategy which will send a header with the time of when I start the flow and valid with a timeout global. obviously, this release strategy will execute until the first message arrived – jandresboyaca Oct 08 '21 at 18:18
  • Yes, it consults the creation time of the group which is, essentially, a reply from the fastest sub-flows. – Artem Bilan Oct 08 '21 at 18:18
  • Is there a way to kwow when a message was trigger into a flow? – jandresboyaca Oct 08 '21 at 18:20
  • See `ChannelInterceptor` abstraction: https://docs.spring.io/spring-integration/docs/current/reference/html/core.html#channel-interceptors. Your `queueFlow` has an input channel as `queueFlow.input`. So, you can declare a `ChannelInterceptor` bean with a `@GlobalChannelInterceptor(patterns = "queueFlow.input")`. However it might be better to start the flow from the `PublishSubscribeChannel` and have another flow subscribed to it to react to the same message differently. – Artem Bilan Oct 08 '21 at 18:26
  • See this JIRA for another option how to handle your use-case: https://jira.spring.io/browse/INT-2208 – Artem Bilan Oct 08 '21 at 19:22