0

I am new to Kafka and I am seeking guidance on how to use Kafka in order to implement the following message pattern:

First, I want the message to be asynchronous and furthermore it needs to be "consumed" i.e. a single consumer should consume it and other consumers won't be able to consume it thereafter.

A use case of this message pattern is when you have multiple instances of a "delivery service" and you want only one of these instances to consume the message (this assumes one cannot leverage idempotency for some reason).

Can someone please advise how to configure the Kafka Consumer in order to achieve the above?

balteo
  • 23,602
  • 63
  • 219
  • 412
  • I found a relevant link: https://medium.com/@ajmalbabu/kafka-0-9-0-clients-db1f43257d30 – balteo Sep 24 '18 at 12:45
  • Any other relevant information is welcomed – balteo Sep 24 '18 at 12:47
  • 1
    I think you've misunderstood what "exactly once" means in Kafka, but messages aren't removed from the topics once read. Consumers don't notify the brokers that they've read the data, only commit the offsets for the consumer group – OneCricketeer Sep 24 '18 at 13:31
  • Thanks a lot for your input @cricket_007 I am going to go though the concepts again. Your comment is interesting. – balteo Sep 24 '18 at 13:40
  • 1
    In a perfect world, messages are never read more than once by a consumer group, but network/application failures occur that can cause the offsets not to be committed, therefore the consumer would be able to re-read messages since it would have to restart from an earlier offset ... I personally haven't played around with the transactions features to know how consumers handle those scenarios otherwise, but you can find the blog series here https://www.confluent.io/blog/enabling-exactly-kafka-streams/ – OneCricketeer Sep 24 '18 at 13:43

3 Answers3

1

I think you're essentially looking to use Kafka as a traditional message queue (e.g. Rabbit MQ) where in the message gets removed after consumption. There has been quite a lot of debate on this. As it is always the case, there are merits and demerits on both sides of the fence.

The answers on this post are more or less against the idea ...

However...

This article talks about an approach on how you could possibly try and make it work. The messages won't really be deleted but the approach is quite similar. It is a fairly comprehensive post that covers the overhead and the optimisations that you could explore to make it more efficient.

I hope this helps!

Lalit
  • 1,944
  • 12
  • 20
1

Great question and its something a lot of us struggle with when deploying and using Kafka. In fact, there are a number of times where a project I was working on tried to use Kafka for the use case you described with very little success.

In a nutshell, there are a few Message Exchange Patterns that you come across when dealing with messaging:

  1. Request->Reply
  2. Publish/Subscribe
  3. Queuing (which is what you are trying to do)

Without digging too deep into why, Kafka was really built simply for Publish/Subscribe. There are other products that implement the other features separately and one that actually does all three.

So a question I have for you is would you be open to using something other than Kafka for this project?

user6135815
  • 13
  • 4
1

You may use spring kafka to do this. Spring Kafka takes care of lot of configurations and boiler plate code. Check example here https://www.baeldung.com/spring-kafka. This should get your started.

Also, you may need to read on how Kafka actually works. The messages that you publish to the Topics in Kafka are natively asynchronous. Your producers don't worry about who consumes it or what happens to the messages once published.

Then consumers in your delivery services should subscribe to the topics. If you want your delivery services to consume a message only once, then the consumers for your delivery services should be in the same group (same group id). Kafka takes care of making sure that the message that was consumed by one of the Consumers (in a same group) won't be available to other Consumers.

The default message retention period is seven days which is configurable in Kafka.

slash
  • 593
  • 8
  • 16