9

I am just exploring Kafka, currently i am using One producer and One topic to produce messages and it is consumed by one Consumer. very simple.

I was reading the Kafka page, the new Producer API is thread-safe and sharing single instance will improve the performance.

Does it mean i can use single Producer to publish messages to multiple topics?

Shankar
  • 8,529
  • 26
  • 90
  • 159
  • Yes, see the docs at http://kafka.apache.org/documentation.html#producerapi. – miguno Aug 23 '16 at 17:54
  • 1
    @miguno I see almost nothing about the question in the link provided. It's about the dependency only :( – Jacek Laskowski Aug 24 '16 at 10:56
  • In the docs above (short section) there's a direct link to the [Java docs of `KafkaProducer`](http://kafka.apache.org/0100/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html), see the sentence "Examples showing how to use the producer are given in the [javadocs](http://kafka.apache.org/0100/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html).". That page has a code example that shows how you can define the destination topic when sending a message. Sorry if I wasn't clear enough. – miguno Aug 24 '16 at 13:03

2 Answers2

9

Never tried it myself, but I guess you can. Since the code for producer and sending the record is (from here https://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html):

Producer<String, String> producer = new KafkaProducer<>(props);
 for(int i = 0; i < 100; i++)
     producer.send(new ProducerRecord<String, String>("my-topic", Integer.toString(i), Integer.toString(i)));

So, I guess, if you just write different topics in the ProducerRecord, than it should be possible.

Also, here http://kafka.apache.org/081/documentation.html#producerapi it explicitly says that you can use a method send(List<KeyedMessage<K,V>> messages) to write into multiple topics.

RadioLog
  • 582
  • 1
  • 4
  • 20
  • FWIW, the latest Kafka docs for the Producer API are at http://kafka.apache.org/documentation.html#producerapi (the link above is for the old 0.8.1 version of Kafka. – miguno Aug 23 '16 at 17:54
  • Where exactly does the producerapi section says about the method? – Jacek Laskowski Aug 24 '16 at 10:55
  • The producerapi section says: "Examples showing how to use the producer are given in the [javadocs](http://kafka.apache.org/0100/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html).". That page has a code example that shows how you can define the destination topic when sending a message. – miguno Aug 24 '16 at 13:07
  • The matter of fact doing so meke the writing process of the producer slow. Has anybody checked the performance of the producer which writes on multiple topics? – Novemberland Mar 19 '19 at 18:21
  • @RadioLog, so can we produce message to multiple topics all having different jaas.conf (keytab and pricipal) ? Pls see my questions here: https://stackoverflow.com/q/58313628/948268 – Kuldeep Jain Oct 11 '19 at 14:45
-1

If I understand you correctly, you are more looking using the same producer instance to send the same/multiple messages on multiple topics.

Not sure about java, but here you can do in C#(.NET) using the Kafka .NET Client DependentProducerBuilder

   using (var producer = new ProducerBuilder<string, string>(config).Build())
        using (var producer2 = new DependentProducerBuilder<Null, int>(producer.Handle).Build())
        {
            producer.ProduceAsync("first-topic", new Message<string, string> { Key = "my-key-value", Value = "my-value" });

            producer2.ProduceAsync("second-topic", new Message<Null, int> { Value = 42 });

            producer2.ProduceAsync("first-topic", new Message<Null, int> { Value = 107 });

            producer.Flush(TimeSpan.FromSeconds(10));
        }
Ajay
  • 1