1

I am trying to implement simple service that pulls messages from kafka wraps them in some data and sends to external service.

What is the common pattern for handling external service unavailabilty when processing message?

So far I am manually commiting messages only when the request to external service was sucesfully. I would like kafka to resend message after some time if it was not committetd so that handling external service failure was transparent to the consumer. I could not find a way to do that though. I am however curious if I am not doing some anti-pattern and there is better solution.

user3364192
  • 3,783
  • 2
  • 21
  • 30

1 Answers1

3

First you need to consider, that Kafka is pull based. Thus, if you want to receive a message a second time, you need to seek() to its offset and poll().

Furthermore, if you want to stop processing messages, you can pause() partitions and later resume() them. See section "Consumption Flow Control" in Consumer JavaDoc: https://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html

Thus, if your external service is down, just pause and wait until it is back.

Matthias J. Sax
  • 59,682
  • 7
  • 117
  • 137