How do I implement multiple producer and multiple consumer in kafka

Question

I am new to kafka, I have requirement like I have lot of server which produce huge amount of log, I want to create multiple producer and consumer.

I have Implemented for single producer and consumer can anyone please help me to understand how I will create multiple producer and consumer.

here is my code for producer and consumer

Producer:-

public class MessageProducerExample {
    
    public static void main(String[] args) {
        
        System.out.println("Hello World");
        
        
         Properties props = new Properties();
         props.put("bootstrap.servers", "localhost:9092");
         props.put("acks", "all");
         props.put("retries", 0);
         props.put("batch.size", 16384);
         props.put("linger.ms", 1);
         props.put("buffer.memory", 33554432);
         props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
         props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

         Producer<String, String> producer = new KafkaProducer<String, String>(props);
         for(int i = 0; i < 100; i++)
             producer.send(new ProducerRecord<String, String>("test", "Msg"+Integer.toString(i),"Msg"+Integer.toString(i)));

         producer.close();
        
    }
}

Consumer :-

public class MessageConsumerExample {
    
    public static void main(String[] args) {
        
        String topicName = args[0];
        
        
         Properties props = new Properties();
         props.put("bootstrap.servers", "localhost:9092");
         props.put("group.id", "test-consumer-group");
         props.put("enable.auto.commit", "true");
         props.put("auto.commit.interval.ms", "1000");
         props.put("session.timeout.ms", "30000");
         props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
         props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
         props.put("auto.offset.reset","earliest");
         
         System.out.println("TopicName="+topicName);
         
         KafkaConsumer<String, String> consumer = new KafkaConsumer<String, String>(props);
         consumer.subscribe(Arrays.asList(topicName));
         
         while (true) {
             ConsumerRecords<String, String> records = consumer.poll(100);
             System.out.println("records"+records.count());
             for (ConsumerRecord<String, String> record : records)
                 //System.out.println("offset = %d, key = %s, value = %s", record.offset(), record.key(), record.value());
                 System.out.println("RECORD_OFFSET"+record.offset()+"RECORD_KEY" +record.key()+"RECORD_VALUE "+record.value());
         }
        }
}

Here is a good thread to answer this doubt: – user3608314 May 18 '22 at 00:48 — user3608314, May 18 '22 at 00:48

score 0 · Answer 1 · edited May 23 '17 at 12:07

0

Kafka producer uses an iothread per broker in async mode. Kafka Producer - By default supports Multithreading?

For consumer, you could create a pool of threads each running a consumer instance. Just make sure that all instances have the same consumer-group. Some examples are given here - Kafka consumer group

edited May 23 '17 at 12:07

Community

1
1

answered Jan 19 '16 at 17:04

Rahul

12,886
13
57
62

How do I implement multiple producer and multiple consumer in kafka

1 Answers1