0

We have springboot app that consumes from a single topic and produces records to multiple topics.

Recently upgraded this app to Sprinboot-2.6.7 and other dependencies accordingly in gradle project.

App is able to consume & produce correctly, BUT the issue is it seems to create kafka adminclients repeatedly(1000s) and seems to be leaking memory(potentially due to this?), ultimately leading to instance crashing and not being able to keep up with lag.

Some kafka related dependent jars in external libraries

   org.apache.kafka:kafka-clients:3.0.1   
   org.springframework.cloud:spring-cloud-stream:3.2.3   
   org.springframework.cloud:spring-cloud-stream-binder-kafka:3.2.3   
   org.springframework.cloud:spring-cloud-stream-binder-kafka-core:3.2.3    
   org.springframework.integration:spring-integration-kafka:5.5.11    
   org.springframework.kafka:spring-kafka:2.8.5

enter image description here

Is there a reason for this? missing configuration?

Gary Russell
  • 166,535
  • 14
  • 146
  • 179
Bhrigu Bhanot
  • 11
  • 2
  • The admin client is only used to provision topics and it is in a try with resources block; as can be seen in the log the client is closed. Your log seems to indicate provisioning activity. It is not clear why you would provision so many bindings in one application. And, what makes you believe this is the source of the memory leak? Have you looked at a heap dump? Please provide an [MCRE](https://stackoverflow.com/help/minimal-reproducible-example) that exhibits this behavior so we can see what is wrong. – Gary Russell Aug 18 '22 at 20:28
  • Your producer log line also has a `3xx` thread number afterwords, so this seems to be related to the AdminClient needing to be created just as many times. – OneCricketeer Aug 18 '22 at 21:17
  • @GaryRussell Found the memory leak issue. After all this, we still had to discard the use of StreamBridge anyway and use the (depricated) StreamListener as we were using with Older springboot version. Streambridge version just could not keep up and the throughput was down from 70GB/5 minute to 45GB/5 min. Which got back to 70GB/5min as soon as StreamBridge was removed. – Bhrigu Bhanot Sep 07 '22 at 14:38

1 Answers1

0

So adminClient was not the issue. The problem was from the default size 10 of the hashmap that stores output channels. I have set spring.cloud.stream.dynamic-destination-cache-size=30, since we have actaully 17 output destinations in app already. In case of the default size 10 of this hashmap "StreamBridge.channelCache" it keeps removing and adding the values to map repeatedly "Once this size is reached, new destinations will trigger the removal of old destinations" calling GC every now and then.

enter image description here

Bhrigu Bhanot
  • 11
  • 2