2

I am newbie to kafka. I have created sample kafka sync producer and consumergroup programs using kafka_2.9.2-0.8.1.1. So My question is, do I need to add multithreading code to producer (like consumergroup class has) to support huge number of requests? I read producer send method is thread safe. So kafka producer will take care of multithreading concepts internally or developer has to code explicitly?

Any help would be highly appreciated.

Thanks, Cdhar

user3401234
  • 31
  • 1
  • 5

1 Answers1

3

There are two types of producers available with Kafka. (1) SyncProducer (2) AsyncProducer. If you set the producer.type configuration as async it will uses the AsyncProducers. By default it uses the Synchronous producer class.

Once running in async mode it creates a separate AsyncProducer instance per broker.And each of these AsyncProducer instances maintains its own internal background thread for sending the messages. These are called ProducerSendThread.

So there is one thread running per broker and your parallelism is based on the number of brokers available in the cluster. So adding new brokers in the cluster should provide you the flexibilities to increase the level of parallelism while producing data using Kafka.But remember adding a new broker to your cluster should be considered taking other paramaters also into consideration.

user2720864
  • 8,015
  • 5
  • 48
  • 60
  • Thanks for the information. As per the Kafka documentation, the latest Kafka 0.8.2 has new producer which is more advanced it seems. Let me try with that. Thank you – user3401234 Feb 20 '15 at 05:52
  • great, would love to have feed back on the same – user2720864 Feb 20 '15 at 07:03
  • to clarify,in case of 1 broker in the cluster(test env) if an async send is held up by a timeout on the broker for instance. Does this mean the whole "async" thread is blocked untill this 1 record has been succesfully delivered? – Havnar Jul 03 '18 at 12:18