2

Scenario :

Stream create [StreamName] --definition " Kafka -zkconnect=10.10.10.1:2181 --topic=<topic name> | MyCompositeModule " --deploy

We are running this stream in distributed mode and redis is the transport bus. Per my understanding, kafka source maintains the offsets for messages consumed by MyCompositeModule (which is a sink as its a module created through 'module compose' process) through [streamname]-kafka-offsets topic. Which is unreadable and I would appreciate if there was a way to read the data from this topic.

Also, when I push the messages from kafka source the messages are queued in redis transport, then the module fetches them from this queue.

If kafka consumer module starts consuming 1000 messages from kafka redis queue- and composite module fails after receiving 10 messages or randomly processed 10 messages.So how to identify remaining 990 [ 1000 ( consumed ) - 10 (processed) = 990 ] unprocessed messages . Even if we check kafka offsets it will show consumed messages count. example: -kafka.offsets - which is unreadable in our process.

So all the unprocessed messages will be in Redis queue as we are using Redis in SpringXD. So can anyone help me out how to identify the unprocessed messages and re sending it to composite module to process.

Basically, I am looking for the recommendations on an elegant solution for robust delivery, adding failure handling capability in spring xd stream when consuming from from kafka source.

hkutluay
  • 6,794
  • 2
  • 33
  • 53
keerthi
  • 58
  • 8

1 Answers1

1

If the messages are effectively consumed from Kafka and moved to the bus, then they will be acknowledged as consumed from the offset manager's perspective.

You can try enabling retry and dead lettering for the Redis Message Bus as described here: http://docs.spring.io/spring-xd/docs/current/reference/html/#error-handling-message-delivery-failures.

Cheers, Marius

Marius Bogoevici
  • 2,380
  • 1
  • 13
  • 14