Using Kafka through Observable(RxJava)

Question

I have a producer (using Kafka), and more than one consumer. So I publish a message in a topic and then my consumers receive and process the message.

I need to receive a response in the producer from at least one consumer (better if it be the first). I'm trying to use RxJava to do it (observables).

Is it possible to do in that way? Anyone have an example?

score 7 · Answer 1 · answered Feb 08 '19 at 20:58

Here is how I am using rxjava '2.2.6' without any additional dependencies to process Kafka events:

import io.reactivex.Observable;

import org.apache.kafka.clients.consumer.Consumer;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;

import java.time.Duration;
import java.util.Arrays;
import java.util.Properties;

...

// Load consumer props 
Properties props = new Properties();  
props.load(KafkaUtils.class.getClassLoader().getResourceAsStream("kafka-client.properties")); 

// Create a consumer
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);

// Subscribe to topics
consumer.subscribe(Arrays.asList(props.getProperty("kafkaTopics").split("\\s*,\\s*")));

// Create an Observable for topic events
Observable<ConsumerRecords<String, String>> observable = Observable.fromCallable(() -> {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofSecond(10);
    return records;
});

// Process Observable events
observable.subscribe(records -> {
    if ((records != null) && (!records.isEmpty())) {
        for (ConsumerRecord<String, String> record : records) {
            System.out.println(record.offset() + ": " + record.value());
        }
    }
});

score 3 · Accepted Answer · edited Apr 25 '19 at 15:48

3

You can use it as follows:

val consumer = new RxConsumer("zookeeper:2181", "consumer-group")

consumer.getRecordStream("cool-topic-(x|y|z)")
  .map(deserialize)
  .take(42 seconds)
  .foreach(println)

  consumer.shutdown()

For more information see: https://github.com/cjdev/kafka-rx

edited Apr 25 '19 at 15:48

Necoras

6,743
3
24
45

answered May 12 '17 at 17:17

Tiago Costa

1,004
4
17
31

1

Is kafka-rx alive? I am considering using it but it has not seen a commit since October 2015. Also, I have never used EPL v.1 license. Not sure if it will be acceptable to my project's sponsor. – Farrukh Najmi Feb 01 '19 at 14:51
2

It seems kafka-rx is using old kafka and Java 8 versions. This solution should not be the accepted answer. The ideal answer should address how to do this using rxjava alone IMO. – Farrukh Najmi Feb 08 '19 at 17:15

score 0 · Answer 3 · answered May 12 '17 at 15:30

Would be better that you'll share your solution first of all...

Since Spring Cloud Stream is a, mh, stream solution, not request/reply, there is no an example to share with you.

You can consider to make your consumers as producers as well. And in the original producer have a consumer to read from the replying topic. Finally you will have to correlate the reply data with the request one.

The RxJava or any other implementation details aren't related.

Using Kafka through Observable(RxJava)

3 Answers3