We are running a java application in Azure Kubernetes and using eventhub messaging libraries and checkpointing for consuming data from azure eventhub.
Application reads data from eventhub and has some processing logic(including processing blob data) and sending multiple messages to destination EH.
We are intermittently getting OutOfMemory, but our application memory utilization is under control in AKS and the application is not stopping the processing.
Scheduler worker in group main failed with an uncaught exception
com.azure.messaging.eventhubs.implementation.PartitionProcessorException: Error in event processing callback
at com.azure.messaging.eventhubs.PartitionPumpManager.processEvents(PartitionPumpManager.java:322) ~[preprocesso r-fullsysinfo-eh-1.10.11.6.jar:?]
at com.azure.messaging.eventhubs.PartitionPumpManager.lambda$startPartitionPump$2(PartitionPumpManager.java:235) ~[preprocessor-fullsysinfo-eh-1.10.11.6.jar:?]
at reactor.core.publisher.LambdaSubscriber.onNext(LambdaSubscriber.java:160) ~[preprocessor-fullsysinfo-eh-1.10. 11.6.jar:?]
at reactor.core.publisher.FluxPublishOn$PublishOnSubscriber.runAsync(FluxPublishOn.java:440) ~[preprocessor-full sysinfo-eh-1.10.11.6.jar:?]
at reactor.core.publisher.FluxPublishOn$PublishOnSubscriber.run(FluxPublishOn.java:527) ~[preprocessor-fullsysin fo-eh-1.10.11.6.jar:?]
at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84) [preprocessor-fullsysinfo-eh-1.10.11.6.jar:?]
at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37) [preprocessor-fullsysinfo-eh-1.10.11.6.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304 ) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:830) [?:?]
Caused by: com.azure.messaging.eventhubs.implementation.PartitionProcessorException: Error in event processing callback
at com.azure.messaging.eventhubs.PartitionPumpManager.processEvent(PartitionPumpManager.java:284) ~[preprocessor -fullsysinfo-eh-1.10.11.6.jar:?]
at com.azure.messaging.eventhubs.PartitionPumpManager.processEvents(PartitionPumpManager.java:318) ~[preprocesso r-fullsysinfo-eh-1.10.11.6.jar:?]
... 11 more
Caused by: java.lang.OutOfMemoryError: Java heap space
We have tried to increase memory limits on AKS nodepool and also increased the JVM.
