Can a Kafka broker retain messages while there are no consumers connected?

Question

I am trying to build a pub/sub application and I am exploring the best tools out there. I am currently looking at Kafka and have a little demo app already running. However, I am running into a conceptual issue.

I have a producer (Java code):

    String topicName = "MyTopic;
    String key = "MyKey";

    Properties props = new Properties();
    props.put("bootstrap.servers", "localhost:9092,localhost:9093");
    props.put("acks", "all");
    props.put("key.serializer","org.apache.kafka.common.serialization.StringSerializer");
    props.put("value.serializer", "org.apache.kafka.common.serialization.ByteArraySerializer");
    Producer<String, byte[]> producer = new KafkaProducer <String, byte[]>(props);

    byte[] data = <FROM ELSEWHERE>;
    ProducerRecord<String, byte[]> record = new ProducerRecord<String, byte[]>(topicName, key, data);

    try {
        RecordMetadata result = producer.send(record).get();
    }
    catch (Exception e) {
        // Nothing for now
    }
    producer.close();

When I start a consumer via the Kakfa command line tools:

kafka-console-consumer --bootstrap-server localhost:9092 --topic MyTopic

and then I execute the producer code, I see the data message show up on my consumer terminal.

However, if I do not run the consumer prior executing the producer, the message appears "lost". When I start the consumer (after executing the producer), nothing appears in the consumer terminal.

Does anyone know if it's possible to have the Kafka broker retain messages while there are no consumers connected? If so, how?

Try to append `--from-beginning` to see if your problem goes away. It's possibly because of the default offset reset strategy. See details on `auto.offset.reset` in the consumer config. — amethystic, Mar 10 '17 at 03:16
@amethystic Yup, that did it! Thanks for the reference. Put this in an answer and I'll accept. Its also nice to see that it displayed all my other "lost" messages as well. — Brett, Mar 10 '17 at 03:23
A very good starting point not to miss these settings during learning is https://kafka.apache.org/quickstart. — randominstanceOfLivingThing, Mar 10 '17 at 03:47

score 7 · Accepted Answer · edited Jun 20 '20 at 09:12

Append --from-beginning to the console consumer command to have it start consuming from the earliest offset. This is actually about the offset reset strategy which is controlled by config auto.offset.reset. Here is what this config means:

What to do when there is no initial offset in Kafka or if the current offset does not exist any more on the server (e.g. because that data has been deleted):

earliest: automatically reset the offset to the earliest offset

latest: automatically reset the offset to the latest offset

none: throw exception to the consumer if no previous offset is found for the consumer's group anything else: throw exception to the consumer.

Can a Kafka broker retain messages while there are no consumers connected?

1 Answers1