Improve performance of the message processing with spring integration dsl

Question

I am using spring integration dsl to receive a message from Kafka and then parse, enrich, and persist in Oracle, Couchbase and publish it to another Kafka topic for downline channels.

Persist and Publish needs to be in a transaction so that all the datasource/datastores are in synch. If Couchbase, Oracle or Publish Kafka topic is unavailable the rollback the transaction.

At the same time I don't want latency in processing the messages as this are real time updates to the business users.

        return IntegrationFlows
                // .from(Jms.messageDrivenChannelAdapter(this.acarsMqListener)) //Get Message
                // from MQ
                .from(org.springframework.integration.jms.dsl.Jms
                        .messageDrivenChannelAdapter(org.springframework.integration.jms.dsl.Jms
                                .container(this.acarsMqConnectionFactory, this.acarsQueue)
                                .transactionManager(transactionManager(this.acarsMqConnectionFactory)).get()))
                .wireTap(ACARS_WIRE_TAP_CHNL) // Log the raw messaged
                .transform(agmTransformer, "parseXMLMessage") // Parse the AGM xml message
                .handle(acarsProcessor, "pushAcarsRawData") // push raw acars data
                .wireTap(ACARS_WIRE_TAP_CHNL_DYNAMODB) // Log the raw messaged
                .transform(agmTransformer, "populateSmi") // Populate SMI
                // .transform(agmTransformer, "populateSmi") //Populate SMI
                .filter(acarsFilter, "filterMessageOnSmi") // Filter on SMI
                .transform(agmTransformer, "populateImi") // Populate IMI
                .filter(acarsFilter, "filterMessageOnSmiImi") // Filter on IMI
                .transform(acarsProcessor, "processEvent") // Parse
                .publishSubscribeChannel(
                        pubSub -> pubSub
                                .subscribe(flow -> flow.bridge(e -> e.order(Ordered.HIGHEST_PRECEDENCE))
                                        .enrichHeaders(
                                                h -> h.headerExpression(KafkaHeaders.MESSAGE_KEY, "payload.flightNbr")) // Add flight number as key
                                        .transform("payload.message") // publish the transformed message
                                        .handle(Kafka.outboundChannelAdapter(kafkaTemplate).topic(acarsKafkaTopic))) // publish to kafka
                                .subscribe(flow -> flow.channel(UPDATE_DATA_STORE_CHNL))) // send to a different channel to update couchbase
                .get();

Can you please suggest what can be done in the integration flow to improve the performance of the processing.

Why do you say “receive from Kafka”, but show JMS? Isn’t your idea to have the whole process in transaction: from receive until publish? — Artem Bilan, Aug 10 '19 at 14:06
yes the message can be received from MQ or Kafka. The idea is to do the whole process in a transaction to rollback in case of any hardware faiiure or exception. Do you see any improvement in the flow that can be implemented to optimize the performance and improve the message processing. — Developer, Aug 10 '19 at 21:55

score 0 · Answer 1 · answered Aug 12 '19 at 13:32

You should profile your application to see where the bottleneck is.

Generally, to improve throughput, you would need to increase concurrency in the listener container to process messages in parallel. With Kafka you will need at least as many partitions as concurrent threads.

Increasing concurrency may not help if the bottleneck is in one of your downstream components; hence the need for profiling.

Improve performance of the message processing with spring integration dsl

1 Answers1