0

I am using Spring Cloud Stream Kafka Binder to produce message into Kafka. I kept producer sync to false, enabled error channel for producer, unclean leader election true on server side.

spring.cloud.stream.kafka.bindings.outputChannelName.producer.sync: false

error-channel-enabled: true

unclean.leader.election.enabled: true

I subscribed to errorChannel, to log the message those are failed to send. I saw for async producer till delivery.timeout.ms reached messages are getting retried for UNKNOWN_TOPIC_OR_PARTITION, NOT_LEADER_FOR_PARTITION, NETWORK_EXCEPTION errors if broker goes down. Not for all errors its retrying. And after delivery.timeout.ms exceeds records are getting expired and reached to errorChannel and printed in logs. But still I have some messages are getting lost, those are not visible in success or errorChannel. I don't find any related exception / error even after log level set to Debug.

Can someone faced same issue earlier?

My Query is:

How to track the lost messages?

Is there any specific configuration I am missing?

Is there any specific exception I need to search in logs?

Any suggestions are welcome.

Ranjit Meher
  • 41
  • 4
  • just a basic question, what is your ack for this. and can you please post your configuration – bananas Sep 19 '22 at 11:27
  • ack=1 spring.cloud.stream.kafka.bindings.output_channel.producer.sync: false spring.cloud.stream.kafka.bindings.output_channel.producer.configuration.retries: 1000 spring.cloud.stream.kafka.bindings.output_channel.producer.configuration.request.timeout.ms: 300000 spring.cloud.stream.bindings.output_channel.producer.error-channel-enabled: true – Ranjit Meher Sep 19 '22 at 13:20
  • acks=0 —the write is considered successful the moment the request is sent out. No need to wait for a response. acks=1 — the leader must receive the record and respond before the write is considered successful. acks=all — all online in sync replicas must receive the write – bananas Sep 19 '22 at 16:30
  • We have one more property on server side unclean.leader.election.enabled: true. So it means if acks=all and replicas is not synced yet and chosen as leader still message can be lost right? we tried acks=all and still messages are getting lost. – Ranjit Meher Sep 19 '22 at 17:26

0 Answers0