Spring Integration Flow DSL with SQS and Reactive

Question

How can I setup a reactive flow using DSL for the following steps:

Receive an SQS Message using SqsMessageDrivenChannelAdapter
Validate the Json message [JsonSchemaValidator class with validate method]
Transform the json to objects
Pass the objects to a service activator (BusinessService : business logic, state machine)
Persist the Objects R2DBC outbound adapter

I was looking at this : https://github.com/spring-projects/spring-integration/blob/master/spring-integration-core/src/test/java/org/springframework/integration/dsl/reactivestreams/ReactiveStreamsTests.java

In the above example, there are dedicated flows created that return a Publisher and in the tests the Publishers are subscribed. However, my flow will be triggered when SqsMessageDrivenChannelAdapter brings in a message into a channel.

How to achieve a reactive flow configuration, for the scenario above steps 1 to 5?

Update : Sample code added

   @Bean
    public IntegrationFlow importFlow()  {
        IntegrationFlows.from(sqsInboundChannel())
                .handle((payload, messageHeaders) -> jsonSchemaValidator.validate(payload.toString()))
                .transform(Transformers.fromJson(Entity.class))
                .handle((payload, messageHeaders) ->businessService.process((Entity) payload))
                .handle(
                        Jpa.outboundAdapter(this.entityManagerFactory)
                                .entityClass(Entity)
                                .persistMode(PersistMode.PERSIST),
                        ConsumerEndpointSpec::transactional)
                .get();
    }

    @Bean
    public MessageProducer sqsMessageDrivenChannelAdapter() {
        SqsMessageDrivenChannelAdapter sqsMessageDrivenChannelAdapter =
                new SqsMessageDrivenChannelAdapter(asyncSqsClient, queueName);
        sqsMessageDrivenChannelAdapter.setAutoStartup(true);
        sqsMessageDrivenChannelAdapter.setOutputChannel(sqsInboundChannel());
        return sqsMessageDrivenChannelAdapter;
    }

    @Bean
    public MessageChannel sqsInboundChannel() {
        return MessageChannels.flux().get();
    }

Update 2 : Moved JPA to a diff thread using executor channel

   @Bean
    public IntegrationFlow importFlow()  {
        IntegrationFlows.from(sqsInboundChannel())
                .handle((payload, messageHeaders) -> jsonSchemaValidator.validate(payload.toString()))
                .transform(Transformers.fromJson(Entity.class))
                .handle((payload, messageHeaders) ->businessService.process((Entity) payload))
                .channel(persistChannel())
                .handle(
                        Jpa.outboundAdapter(this.entityManagerFactory)
                                .entityClass(Entity)
                                .persistMode(PersistMode.PERSIST),
                        ConsumerEndpointSpec::transactional)
                .get();
    }

    @Bean
    public MessageProducer sqsMessageDrivenChannelAdapter() {
        SqsMessageDrivenChannelAdapter sqsMessageDrivenChannelAdapter =
                new SqsMessageDrivenChannelAdapter(asyncSqsClient, queueName);
        sqsMessageDrivenChannelAdapter.setAutoStartup(true);
        sqsMessageDrivenChannelAdapter.setOutputChannel(sqsInboundChannel());
        return sqsMessageDrivenChannelAdapter;
    }

    @Bean
    public MessageChannel sqsInboundChannel() {
        return MessageChannels.flux().get();
    }

    @Bean
    public MessageChannel persistChannel() {
        return MessageChannels.executor(Executors.newCachedThreadPool()).get();
    }

and what code have you written? – Toerktumlare Mar 09 '21 at 15:28 — Toerktumlare, Mar 09 '21 at 15:28
I have updated the original post with the code – Nikhil Mar 09 '21 at 16:03 — Nikhil, Mar 09 '21 at 16:03

score 1 · Accepted Answer · answered Mar 09 '21 at 15:28

1

You probably need to make yourself more familiar with what we have so far for Reactive Streams in Spring Integration: https://docs.spring.io/spring-integration/docs/current/reference/html/reactive-streams.html#reactive-streams

The sample you show with that test class is fully not relevant to your use case. In that test we try to cover some API we expose in Spring Integration, kinda unit tests. It has nothing to do with the whole flow.

Your use-case is really just a full black box flow starting with SQS listener and ending in the R2DBC. Therefore there is no point in your flow to try to convert part of it into the Publisher and then bring it back to another part of the flow: you are not going to track some how and subscribe to that Publisher yourself.

You may consider to place a FluxMessageChannel in between endpoints in your flow, but it still does not make sense for your use-case. It won't be fully reactive as you expect just because a org.springframework.cloud.aws.messaging.listener.SimpleMessageListenerContainer is not blocking on the consumer thread to be ready for a back-pressure from downstream.

The only really reactive part of your flow is that R2DBC outbound channel adapter, but probably it does not bring you too much value because the source of data is not reactive.

As I said: you can try to place a channel(channels -> channels.flux()) just after an SqsMessageDrivenChannelAdapter definition to start a reactive flow from that point. At the same time you should try to set a maxNumberOfMessages to 1 to try to make it waiting for a free space in before pulling the next mesasge from SQS.

answered Mar 09 '21 at 15:28

Artem Bilan

113,505
11
91
118

I have added a sample code in the post. In the above flow, since `sqsInboundChannel` is a flux channel, it should start a reactive flow and then each `.handle` is chained to the `sqsInboundChannel` Is that correct?. In this scenario, i.e above code, should each handler accept and return a `Mono`? For example, `jsonSchemaValidator.validate` ? Also currently in the above code there are 2 non-reactive components - `SimpleMessageListenerContainer` and JPA outbound adapter. Until, these two are replaced by their reactive counterparts, is the above flow definition correct? – Nikhil Mar 09 '21 at 16:01
No, it's OK to have an in-memory reply handlers as regular. They are called from kinda `map()` operation of the `Flux`. The JPA part is something new and that's definitely a different story. You may fail Blockhound inspection if you don't sort that channel adapter to a different thread. Fortunately it is easy with an `ExecutorChannel` in Spring Integrtion. On the other hand if it is not reactive over there, there is probably no fully point to pursue reactivity since you have it only in the middle where you do a logic in the memory anyway, so that could happen on an SQS thread. – Artem Bilan Mar 09 '21 at 16:16
Thanks ! I have added an `ExecutorChannel`, in Update 2 of this post. Also, how will `maxNumberOfMessages` to `1` ? Will it the `sqs adapter` poll 1 message at a time from the queue, trying to understand this. – Nikhil Mar 09 '21 at 16:37
Right. One message at a time. To not pull too much to the memory before you start processing them. Just because there are too many thread shifting in the `SimpleMessageListenerContainer` for SQS, I would be as careful as possible to not lose data in between. – Artem Bilan Mar 09 '21 at 16:43
Is there no impact of polling 1 message at a time? Is this at a thread level? For example, if I have 100 messages in the queue, will there be 100 polls using s single thread? What about long polling, in this case I can poll only 1 message until `waitTimeOut`? Sorry for basic questions and if this going off-topic – Nikhil Mar 09 '21 at 17:00
I think it is going. Looking to the source code of the `SimpleMessageListenerContainer` for SQS, I would fully stay away of it. It does schedule every single message to the task executor and comes back for more messages pulling. So, there is very high risk to lose messages. – Artem Bilan Mar 09 '21 at 17:30

Spring Integration Flow DSL with SQS and Reactive

1 Answers1