0

I have a concern regarding the Debizum outbox pattern ,that is when the Kafka connect consumes messages from outbox table and trying to produce to kafka topic ,if the kafka brokers are down does the debizum retries the message delivery or just fail?

Do we have any configuration for kafka produce fail scenario?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Shiraz Kamal
  • 1
  • 2
  • If the output table is persistently stored, and the connector maintains its "progress" (offset/timestamp/row), then it shouldn't matter if Kafka is down as the data is buffered in the table. – OneCricketeer Apr 05 '22 at 19:51
  • Yes outbox table is a persistant storage ,and the connector maintains its "progress" (offset/timestamp/row), means what? do we need to do some configuration on connector side? – Shiraz Kamal Apr 06 '22 at 06:44
  • The whole point of the outbox table is an intermediate database before writing to Kafka. Databases (usually) persist data. If Kafka is down, the data is still able to be read from the database. It's the connectors responsibility to maintain which ID/timestamp was last read successfully. Connect will fail if Kafka is down, not just Debezium, regardless of the outbox pattern. You can read the Debezium blog for more information https://debezium.io/blog/2019/02/19/reliable-microservices-data-exchange-with-the-outbox-pattern/ – OneCricketeer Apr 06 '22 at 15:28

0 Answers0