0

I create a flow, which polls rows from database by status, validate them and after that aggregate to collection. After processing the entire flow, each line is set to the appropriate status. But when I use aggregator with release strategy TimeoutCountSequenceSizeReleaseStrategy, and elapsed time is so small, release group doesn't happen. And after that the following polling occurs in another thread, but previous message group wasn't handled until amount of messages won't reach the target(threshold) in the strategy.

Code of my flow:

@Bean
public IntegrationFlow testFlow(EntityService entityService,
                                EntityValidator entityValidator,
                                EntityFlowProperties properties,                                       
                                EntityChecker checker) {
    return IntegrationFlows
            .from(getMessageSource(entityService::getByStatus, properties.getMaxRowsPerPoll()),
                    e -> e.poller(getPollerSpec(properties)))
            .split()
            .transform(entityValidator::validate)
            .filter(ValidationStatus<Entity>::isValid, filter ->
                    filter.discardFlow(flow -> flow.handle(entityService::handleValidationErrors)))
            .transform(ValidationStatus<Entity>::getEntity)
            .aggregate(aggregatorSpec -> aggregatorSpec.releaseStrategy(new TimeoutCountSequenceSizeReleaseStrategy(5, 10000)))
            .transform(checker::checkOnSomething)
            .split()
            .transform(CheckResultAware<Entity>::getEntity)
            .handle(entityService::saveAndChangeStatus)
            .get();

I expect to perform aggregation on the same thread as polling, and don't make a new polling until the current flow is over.

The way of changing statuses between polling and aggregation isn't suitable.

Is there a way to do this?

Roman
  • 61
  • 1
  • 6

1 Answers1

0

Why do you need TimeoutCountSequenceSizeReleaseStrategy; your sequences are finite; just use the default SimpleSequenceSizeReleaseStrategy.

However the TimeoutCountSequenceSizeReleaseStrategy should release based on the sequence size anyway.

But, it's not really suitable for your use case because you can be left with a partial group in the store.

Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • Why that `messages.isComplete()` is not enough to understand that sequence has been fullfiled? I think the logic is correct, but there is need some optimization to be done: we don't need to look for an `earliestTimestamp` if group is complete already. – Artem Bilan Apr 24 '19 at 13:48
  • The group **is** released when the group is complete. The problem is with a finite sequence, say 10 items; let's say when item #6 is processed, the timeout occurs, releasing 6; then 4 more are delivered and they will be orphaned. – Gary Russell Apr 24 '19 at 13:53
  • Well, that's why a `groupTimeout` is better, so those orphans are going to be partially released eventually anyway. I even wonder now if we should deprecate this `TimeoutCountSequenceSizeReleaseStrategy` in favor of `groupTimeout` on the aggregator. Just because it has this vulnerability to never release the rest of group. Although there is still `MessageGroupStoreReaper`... – Artem Bilan Apr 24 '19 at 14:58
  • Yes; it's really been superseded by the group timeout. – Gary Russell Apr 24 '19 at 15:02
  • But even with a groupTimeout, if there are more records, new records will be processed in another thread. The question was another, can we stop polling until the current records are processed in aggregation(by timeout) and the flow is complete? – Roman Apr 25 '19 at 08:59
  • As I said, for your use case you should not be using a timed release strategy, use the default. – Gary Russell Apr 25 '19 at 13:30