0

About application.

We are running a Kafka streaming application in Azure Kubernetes Service (AKS). Which consumes message from Azure Event hubs, performs business logic and writes output message to another event hub. We store our configurations in Azure App Configuration Service and read all configs at once and cache it in an object and reuse it when required.

Jar azure-data-appconfiguration:jar:1.2.4 is used to connect to App Configuration, which transitively download the jar io.netty:netty-buffer:jar:4.1.68.Final:compile . Below is the dependency tree

+- com.azure:azure-data-appconfiguration:jar:1.2.4:compile
|  +- com.azure:azure-core:jar:1.22.0:compile
|  \- com.azure:azure-core-http-netty:jar:1.11.2:compile
|     +- io.netty:netty-buffer:jar:4.1.68.Final:compile

Please note: As part of businees logic we apply Changed file to Base file to generate new files. We use class "https://github.com/google/diff-match-patch/blob/master/java/src/name/fraser/neil/plaintext/diff_match_patch.java"

ISSUE

The issue we are facing is memory utilization is increasing gradually for this streaming application. When We took heap dump for analysis We found that object io.netty.buffer.PooledByteBufAllocator is occupying more space we suspect leak here. enter image description here We can not exclude netty-buffer because it is used to connect to Azure App Configuration.

What We tried we upgraded the jar azure-data-appconfiguration to latest version 1.3.6 thinking may be there may be memory leak in older version, But still same issue persist. below is the dependency tree.

+- com.azure:azure-data-appconfiguration:jar:1.3.6:compile
|  +- com.azure:azure-core:jar:1.31.0:compile
|  \- com.azure:azure-core-http-netty:jar:1.12.4:compile
|     +- io.netty:netty-buffer:jar:4.1.79.Final:compile

Support required

  1. Is there any known memory leak in netty-buffer jar?
  2. Is there any way to know stack trace where these objects are getting created.
  3. Please suggest best way to debug this leak and fix.
chandu ram
  • 251
  • 2
  • 5
  • 19

2 Answers2

0

This seems to be a possible issue with the SDK. An issue has been created on the GitHub repo to track see: https://github.com/Azure/azure-sdk-for-java/issues/30666

mrm9084
  • 430
  • 1
  • 3
  • 12
  • Hello, mrm9084, Thank you very much for your response. I can see some update in the ticket that is is fixed as part of the ticket. https://github.com/Azure/azure-sdk-for-java/pull/30670. So In which version of the SDK we can expect the fix? – chandu ram Aug 27 '22 at 12:24
0

I've created a simple AppConfiguration based application which created 500 keys, listed them, and then deleted them while taking periodic heap dumps and I didn't find any leaks. The mentioned GitHub issue (https://github.com/Azure/azure-sdk-for-java/issues/30666) was for a code path that an AppConfiguration client won't use.

Based on the heap dump image posted this looks to be normal memory behavior. PooledByteBufAllocator will allocate 2 "arenas" per thread and each arena will allocate a 4MB chunk, which if the machine being used is 2 threads then that's the 16MB highlighted. See this PR for more information: https://github.com/netty/netty/pull/12108

Alan Zimmer
  • 1
  • 2