2

I am having an rest post endpoint which consumes data and writes to Kafka using Spring Cloud Stream Kafka Binder. Right now we are not having any error handling in place. But we want to make this endpoint fault tolerant by adding an extra check whenever data is not written to Kafka. We intend to send an exception info object when data is not written to Kafka. I am trying to achieve this using global errors in this way

@ServiceActivator(inputChannel = "errorChannel")
public void handle(final ErrorMessage em) {
logger.error("encountered exception" + em.getOriginalMessage().toString());
throw new RuntimeException(em.getOriginalMessage().toString);
}

My doubt is two fold:

  1. Is this the correct way to handle exceptions when the data we write to Kafka fails.
  2. Is this handle method called whenever data write is failed and is this change propagated to system level error handling.

If there is another process please suggest. We are currently exploring application level error handling and global level error handling. System level error handling is off the table for now. Thanks in advance.

Teja M
  • 39
  • 8

1 Answers1

2

You have to opt in for async error handling, using errorChannelEnabled https://docs.spring.io/spring-cloud-stream/docs/3.1.3/reference/html/spring-cloud-stream.html#_producer_properties

You can get an exception on the sending thread, by setting the kafka producer property sync to true https://docs.spring.io/spring-cloud-stream-binder-kafka/docs/3.1.3/reference/html/spring-cloud-stream-binder-kafka.html#kafka-producer-properties

Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • This is quite helpful. I have one more question. By enabling the sync property can I enclose the code snippet for sending the data in a try catch block and propagate the exception like we do in the normal scenarios. – Teja M May 27 '21 at 16:55
  • Yes; that is correct; with sync, the channel adapter waits for the send future to complete (normally, or with an exception, which can be caught). – Gary Russell May 27 '21 at 16:58
  • We are not using futures here we are using Message channel send to send the data. So can we wrap it in try catch block and handle it. Sorry for the dumb question please don't mind. – Teja M May 27 '21 at 18:05
  • The future (returned by a `KafkaTemplate`) is used downstream in the channel adapter; when sync is true, the adapter waits for it to complete (on the calling thread) so you will get an exception if the send fails). – Gary Russell May 27 '21 at 18:09
  • Thanks Gary, this is really helpful. If I want to do this asynchronously I can do it with the code snippet I mentioned and I need to keep errorChannel enabled right. This way whenever errorchannel gets a message it will throw an exception right. – Teja M May 27 '21 at 18:51
  • 1
    It won't "throw" an exception, the method will get an `ErrorMessage` with the exception in the payload; throwing an exception there will not help you because it is a framework thread that calls it; there is no way to propagate an async exception back to the controller, unless you do it yourself (e.g. via a blocking queue). If you want the exception propagated to the rest controller, `sync` is the best option. – Gary Russell May 27 '21 at 19:04
  • Thanks a lot Gary. Appreciate your help. – Teja M May 27 '21 at 19:57
  • Gary, one more doubt. If switch from spring cloud stream kafka to spring kafka can I catch the error and propagate it to the controller using futures. I don't want to lose the ability of asynchronous execution. Hence can you please advice on how to proceed. Thanks in advance. – Teja M Jun 03 '21 at 12:06
  • Yes, the `KafkaTemplate.send()` operations return a `Future` which you can wait on for completion. When successful, it returns metadata (partition, offset) where the record was written to. – Gary Russell Jun 03 '21 at 13:31