19

I have kafka producer for my java based web application to push messages to Kafka. As per the documentation I could see kafka producer is thread safe. Does it mean that I can have single instance of Kafka producer and use it by different threads ( web requests ) each will open and close the producer in my case. Will this create any issues ? Or Is better to initiate Producers per request ?

Bill Goldberg
  • 1,699
  • 5
  • 26
  • 50

2 Answers2

21

Yes, KafkaProducer is threadsafe.
Refer to Class KafkaProducer

A Kafka client that publishes records to the Kafka cluster.

The producer is thread safe and should generally be shared among all threads for best performance.

The producer manages a single background thread that does I/O as well as a TCP connection to each of the brokers it needs to communicate with. Failure to close the producer after use will leak these resources.

Shawn Guo
  • 3,169
  • 3
  • 21
  • 28
  • 1
    I am worndering if it handle the below scenario.. One thread opens a producer and before sending data another thread closes it . Will there be exception when execute send by first thread? – Bill Goldberg Mar 24 '16 at 06:53
  • 2
    absolutely exception is thrown. usually we use micro-container like spring context, and close kafka consumer only when destroying spring context. – Shawn Guo Mar 24 '16 at 10:16
  • 1
    If only has one instance of `KafkaProducer` then where and how can we call `close` method on it? – ttt Jul 26 '18 at 06:45
  • Is this also true for the new transactional APIs? The documentation doesn't specify, and I'm seeing behavior that indicates it is not safe. – Raman Oct 06 '19 at 04:01
  • @Raman i had the same questionI. think for multi-threaded case with transactions, a ThreadLocal Producer with unique transactionId for each producer should be used – Ashika Umanga Umagiliya Jun 10 '21 at 07:12
  • 3
    @AshikaUmangaUmagiliya yes I concluded that the transactional APIs are not thread-safe. I actually find the design of them poorly done -- they assume a certain consumer threading context which could have been avoided with the use of transaction handles. – Raman Jun 10 '21 at 13:54
3

By far the best approach (which is typical of most stateful clients connectors, eg SQL clients, elasticsearch client, etc) is to instantiate a single instance at application start and share it across all threads. It should only be closed on application shutdown.

ppearcy
  • 2,732
  • 19
  • 21
  • Hello, I'm beginning in java and I'm wondering how to do such things, do you have any examples I could look at? thanks – Simon Jan 22 '21 at 08:39