0

We have a web application with java8+Springboot+Spring Integration. We are using the DirectChannels for message transfers within our application. There are two inboundadapter to this application a-SOAP web service b- MQ, Also we have outbound adapter calls to two applications :

  1. For auditing we have another application which we call via Rest api.
  2. The request recieved by our application is sent to x application these are rest calls.

For x application we are doing httpConnection pooling and we have defaultMaxPerRoute value as 100 and maxTotal value as 200.

Problem :

we are doing a load test of our application using 60 users for around 30 mins. intially the application is processing the message really quickly. The request is within application for max 1sec before it makes the call to x application above.

However what we are observing is that after some point the app becomes really slow even simple method execution for a particular request is taking around 2 seconds and the request remains in the application for around 10-15 seconds which is hampering the performance of the application. This is observed for all requests not specific to any type or api.

We are not able to understand what is causing this slowness. We have added multiple loggers into our application but still not getting any clue. We even checked applciation logs there is no Out of memory error or there is no CPU getting fully utilized.

It will be great if anyone can share some input on what could be wrong or what else should we check.

Below is the logs statement for one of the request which is taking time. I have added loggers at the start and end of each channel. It is evident that even when passing the request from one channel to another it is taking around 2 seconds. Also a simple transformer is taking 2 seconds. So I am not able to identify the root cause for this.

channel 1
2022-11-22 09:24:51 Transformer [INFO] converted the Source object to String xml for message 
2022-11-22 09:24:54 AuditService [DEBUG] Completed first channel for : 


channel 2
2022-11-22 09:24:56 AuditService[DEBUG] Creating entry audit for operation
2022-11-22 09:24:59 AuditService [DEBUG] log just before pre rest channel call

channel 3 
2022-11-22 09:25:02 AuditService [DEBUG] starting rest pre channel 

2022-11-22 09:25:05 RequestTransformer [DEBUG] About to process request 

2022-11-22 09:25:05 RequestTransformer [DEBUG] The json after conversion 



channel 4
2022-11-22 09:25:07 AuditService [DEBUG] Completed request sent audit for operation
2022-11-22 09:25:08 AuditService[DEBUG] Completed request sent audit for operation
Ashish Kathait
  • 226
  • 6
  • 15
  • I'd suggest to start disabling some of your services to spot a bottleneck. For example I think your "auditing we have another application which we call via Rest api" is guilty. Just turn this option off and you should see a difference. Does it really required to be called within the current request thread? Cannot you put your auditing into some `QueueChannel` to let it to be processed on background in the other thread? – Artem Bilan Dec 09 '22 at 12:58
  • @ArtemBilan I checked Datadog but i don't see that auditservice is taking more time. – Ashish Kathait Dec 19 '22 at 15:19
  • As you wish. I'm not familiar with any profiling tools. That's why I suggest to disable services or one by one or all together and turn on one by one to spot the culprit. The audit service might not take too much time by itself, but requests might wait for free thread in the pool. And that might block your application process. – Artem Bilan Dec 19 '22 at 15:26

1 Answers1

0

I found the root cause of this problem. The problem was in the spring core library. We were using the version : 4.3.5 In our code we were using @Header annotation which was triggering the Spring GenericConversionService where there is a converterCache getting created and there was always a cache miss happening because the equals method of TypeDescriptor class had faulty logic for Annotations. Because of this when we were performing the load test, with the increase in the request the performance was degrading rapidly due to increase in Map size and cache miss everytime.

github url for this issue : https://github.com/spring-projects/spring-framework/issues/19626

commit url for fix https://github.com/spring-projects/spring-framework/commit/e38c020535dc5d6b326d5e22f8a59dc35ba7361a:

  • Fix 1 : Update Spring version to 4.3.6 or above.

  • Fix 2 : I removed reference of @Header annotation as they were few and i could get the header value using the Message object of Spring integration(We also upgraded to the latest version of Spring and SpringBoot later after issue identification).

Ashish Kathait
  • 226
  • 6
  • 15