How to use Multithreading in Kafka

Question

I am trying to read kafka messages from producer using Java Multithreading.

Suppose, Kafka Producer send multiple messages to Kafka Consumer. then how to read those multiple messages seperatly using ExecutorService in JAVA

this is not a question about kafka more about how to use java executor service or multithreading api. The executor service you can use a Future or a Threadpool depending on your taste each thread can then subscribe to a topic and stream the messages. You should describe more what you really want to do. — Hamuel, May 18 '18 at 08:19
This could be the best way to do it instead of using ExecutorService https://stackoverflow.com/questions/58889112/spring-kafkalistener-and-concurrency — Player_Neo, Feb 06 '20 at 08:42

serkan kucukbay · Answer 1 · 2018-05-24T08:41:52.500

I have implemented your mentioned case and i am going to share steps that you should follow.

Create a consumer class which must implement Runnable interface, it must have a Kafkaconsumer instance as a class member. You can configure consumer properties in constructor method.

public LogConsumer(List<String> topics, String group, String brokerList) {

    Properties propsConsumer = new Properties();
    propsConsumer.put("bootstrap.servers", brokerList);
    propsConsumer.put("group.id", group);
    propsConsumer.put("enable.auto.commit", "false");
    propsConsumer.put("key.deserializer", StringDeserializer.class);
    propsConsumer.put("value.deserializer", ByteArrayDeserializer.class);
    propsConsumer.put("auto.offset.reset", "latest");

    this.consumer = new KafkaConsumer(propsConsumer);
    this.consumer.subscribe(topics);
}

Then, in run method you can consume each kafka message as shared below.

public void run() {
    try {
        while (!flagOfThread) {
            ConsumerRecords<String, byte []> records = consumer.poll(10000);
            for (ConsumerRecord<String, byte []> record : records) {
                handleRecord(record);
            }
            // At least one
            consumer.commitSync();
        }
    } catch (WakeupException e) {
        // Ignore exception if closing
        if (!flagOfThread){
            LOG.error("Log Consumer is shutting down.",e);
        }

    } finally {
        consumer.close();
    }
}

In your application runner class, you should create a threadpool, this thread pool count must be equal to topic partition count .

 ExecutorService executorService = Executors.newFixedThreadPool(parallelismCount);
    for (int i = 0; i < parallelismCount; i++) {
        ExecutionLogConsumer bean = new LogConsumer(/*parameters*/);
        executorService.execute(bean);
    }

Now you can start to consume your messages from kafka :)

How to use Multithreading in Kafka

1 Answers1