0

Wondering if this is possible? I'd like to have 2 debezium connector processes (with different server ids) pointing to the same MySQL database and receiving the data as if they are workers sharing the load. Currently I observe the behaviour of pub/sub where both gets a copy of the same event. If I use the same server.id, then the first process dies complaining that the process with the same server id has been started.

I use Kafka as a store.

Wondering if this is possible or Debezium connectors are always meant to be "single process" and not horizontally scale?

user1189332
  • 1,773
  • 4
  • 26
  • 46

1 Answers1

0

Just a thought here. MySql DB delta bin log must be read sequentially by a single task.To distribute the tasks against multiple tables use multiple connectors reading from different tables from the same DB

https://debezium.io/documentation/reference/stable/connectors/mysql.html

tasks.max : 1: The maximum number of tasks that should be created for this connector. The MySQL connector always uses a single task and therefore does not use this value, so the default is always acceptable.

Ref : https://debezium.io/documentation/reference/stable/connectors/mysql.html

You could have HA using same group id , when one of the connect servers go down the other connector would pick up where the previous connector left off.

Ramachandran.A.G
  • 4,788
  • 1
  • 12
  • 24