I have two apps, one is "main" app which runs on one specific server and another type which runs on some tens/hundreds of other servers and usually communicates with the main app (two way). There is a lot of shared data between the main app the other instances which should be writable from both sides (some of the postgres tables). Every instance including the main can be offline sometimes while others should still be able to work and manipulate data persistently. Data from the main app should sync to the others and data from the others should sync to the main app (possibly when the counter part goes online after being offline for some time).
I was considering running kafka server on each app with debezium so it captures all writes to local DB and also some sort of kafka-connect on each app to consume the remote kafkas. Main app will have to consume from all others and all other apps will consume from main.
Biggest thing I'm uncertain about is how can such thing handle conflicts, because essentially this is a multi master replication. I was wondering if anybody have had similar experience or can add insights about such a design.
