My java web application is running in docker container. Java application is sending messages to Kafka (~ 300 msgs/hour). The server is crashing very few hours. I need tot restart the server every few hours at this time. I see that the number of open files are continuesly increasing. The output of below command is increasing ~ 50/sec
lsof | grep kafka | wc -l
I have tried Kafka versions from 2.4.0 to 2.6.0. (both server and Kafka-client java) yet the problem persists.
**Java code **
public class KafkaProducer {
private static KafkaProducer single_instance = null;
public Producer<String, KafkaRequest> kafkaProducer = null;
private KafkaProducer(String url)
{
Properties props = new Properties();
props.put("bootstrap.servers", url);
kafkaProducer = new org.apache.kafka.clients.producer.KafkaProducer<>(props, new StringSerializer(), new KafkaRequestSerializer());
}
public static KafkaProducer getInstance(String url)
{
if (single_instance == null)
single_instance = new KafkaProducer(url);
return single_instance;
}
}
And sending message
public void postKafkaMessage(String topicName, KafkaRequest kafkaRequest, String key, String kafkaUrl) {
KafkaProducer kafkaProducer = KafkaProducer.getInstance(kafkaUrl);
kafkaProducer.kafkaProducer.send(new ProducerRecord<>(topicName, key, kafkaRequest));
}
Sample lsof snippets
**COMMAND PID TID TASKCMD USER FD TYPE DEVICE SIZE/OFF NODE NAME**
java 23 1811 FlowRunne root 210u IPv4 95985 0t0 TCP 7adf829e2db1:37696->kafka.server_elk:9092 (ESTABLISHED)
java 23 1811 FlowRunne root 211u sock 0,8 0t0 534942 protocol: TCP
java 23 1811 FlowRunne root 212u sock 0,8 0t0 534943 protocol: TCP
java 23 1811 FlowRunne root 213r FIFO 0,11 0t0 96862 pipe
java 23 1811 FlowRunne root 214w FIFO 0,11 0t0 96862 pipe
java 23 1811 FlowRunne root 215u a_inode 0,12 0 8697 [eventpoll]
java 23 1811 FlowRunne root 216u sock 0,8 0t0 531973 protocol: TCP
java 23 1811 FlowRunne root 217u IPv4 96864 0t0 TCP 7adf829e2db1:37718->kafka.server_elk:9092 (ESTABLISHED)
java 23 1811 FlowRunne root 218u sock 0,8 0t0 534027 protocol: TCP
java 23 1811 FlowRunne root 219r FIFO 0,11 0t0 95990 pipe
java 23 1811 FlowRunne root 220w FIFO 0,11 0t0 95990 pipe
java 23 1811 FlowRunne root 221u a_inode 0,12 0 8697 [eventpoll]
java 23 1811 FlowRunne root 222u sock 0,8 0t0 534028 protocol: TCP
java 23 1811 FlowRunne root 223u IPv4 95992 0t0 TCP 7adf829e2db1:37728->kafka.server_elk:9092 (ESTABLISHED)
java 23 1811 FlowRunne root 224u sock 0,8 0t0 534030 protocol: TCP
java 23 1811 FlowRunne root 225r FIFO 0,11 0t0 95994 pipe
java 23 1811 FlowRunne root 226w FIFO 0,11 0t0 95994 pipe
java 23 1811 FlowRunne root 227u a_inode 0,12 0 8697 [eventpoll]
java 23 1811 FlowRunne root 228u sock 0,8 0t0 534032 protocol: TCP
java 23 1811 FlowRunne root 229u IPv4 98150 0t0 TCP 7adf829e2db1:37738->kafka.server_elk:9092 (ESTABLISHED)
java 23 1811 FlowRunne root 230u sock 0,8 0t0 534944 protocol: TCP
java 23 1811 FlowRunne root 231u sock 0,8 0t0 532895 protocol: TCP
java 23 1811 FlowRunne root 232r FIFO 0,11 0t0 98838 pipe
java 23 1811 FlowRunne root 233w FIFO 0,11 0t0 98838 pipe
java 23 1811 FlowRunne root 234u a_inode 0,12 0 8697 [eventpoll]
java 23 1811 FlowRunne root 235u sock 0,8 0t0 534035 protocol: TCP
java 23 1052 pool-8-th root 434u IPv4 111060 0t0 TCP 7adf829e2db1:38250->kafka.server_elk:9092 (ESTABLISHED)
java 23 1052 pool-8-th root 440u IPv4 112668 0t0 TCP 7adf829e2db1:38260->kafka.server_elk:9092 (ESTABLISHED)
java 23 1052 pool-8-th root 446u IPv4 113865 0t0 TCP 7adf829e2db1:38270->kafka.server_elk:9092 (ESTABLISHED)
java 23 1052 pool-8-th root 452u IPv4 111071 0t0 TCP 7adf829e2db1:38280->kafka.server_elk:9092 (ESTABLISHED)
java 23 1052 pool-8-th root 458u IPv4 113871 0t0 TCP 7adf829e2db1:38290->kafka.server_elk:9092 (ESTABLISHED)
java 23 1052 pool-8-th root 464u IPv4 113876 0t0 TCP 7adf829e2db1:38296->kafka.server_elk:9092 (ESTABLISHED)
java 23 1052 pool-8-th root 470u IPv4 111080 0t0 TCP 7adf829e2db1:38308->kafka.server_elk:9092 (ESTABLISHED)
java 23 1052 pool-8-th root 476u IPv4 112964 0t0 TCP 7adf829e2db1:38328->kafka.server_elk:9092 (ESTABLISHED)
java 23 1052 pool-8-th root 483u IPv4 111102 0t0 TCP 7adf829e2db1:38340->kafka.server_elk:9092 (ESTABLISHED)
java 23 464 kafka-pro root 1189u sock 0,8 0t0 614811 protocol: TCP
java 23 464 kafka-pro root 1190u sock 0,8 0t0 622360 protocol: TCP
java 23 464 kafka-pro root 1191u sock 0,8 0t0 536231 protocol: TCP
java 23 464 kafka-pro root 1192u sock 0,8 0t0 622361 protocol: TCP
java 23 464 kafka-pro root 1193u sock 0,8 0t0 536234 protocol: TCP
java 23 464 kafka-pro root 1194u sock 0,8 0t0 536235 protocol: TCP
java 23 464 kafka-pro root 1195u sock 0,8 0t0 616746 protocol: TCP
java 23 464 kafka-pro root 1196u sock 0,8 0t0 616747 protocol: TCP
