Does java KafkaProducer open any pipes or sockets to connect to kafka? What happens if kafka producer object is not able to connect to kafka, does it close these pipes and sockets?
1 Answers
Every KafkaProducer
has to open TCP sockets locally to attempt to connect to the the Kafka servers that it was configured with. It needs to do this at instantiation time, because Kafka will fetch metadata from the servers (which topics are available, which are the leaders for every partition, whether to compress data or not, etc), so it's ready to go whenever there's a producer.send()
request. Assuming the Kafka servers are up and reachable, there will be remote TCP sockets on those servers accepting the connections, and after a handshake process, they will settle on a ESTABLISHED state on both sides. Depending on the Kafka version, and on the type of producer, there will also be other TCP connection requests being sent to zookeeper services, and optionally even to schema registry servers.
If the Kafka/Zookeeper/other servers are not available, for whatever reason, it will still need to open those TCP sockets locally, to initiate the request, but they would never get to the ESTABLISHED state. Assuming there is a retry loop, the sockets might be reused or new ones might be established, until either the connection is established, or the connection request is cancelled.
So, to more directly answer your question, assuming the connection request is cancelled if the KafkaProducer is not able to talk to Kafka, yes, it should clean up after itself and close all the sockets it opened. As far as I know, Kafka clients never uses Unix pipes to communicate, the only way to reach a Kafka server, to either produce or consume data, is throught TCP connections.

- 16,372
- 11
- 56
- 73
-
Thanks for the detailed explanation, but i observed that it was openning pipes also, will double check. The reason for asking this was kafka producer was not closing the connections and hence we are having file descriptors leak, causing too many files open issue. Now we got the root cause, it was slf4j version miss match. in the close function of producer there is a log message, and it requires latest version of slf4j, so no such method exception was thrown and the connections were not closed. This we observed after asking this question in this forum. Thanks. – rajukemburi Aug 09 '19 at 12:06