We need to set up a kafka cluster with L nodes (with L kraft controllers). There are N producers and N + M consumers and we are interested only in "live" data.
- N consumers are classified as 'local' so they are intended to receive data from their local producer
- M consumers are classified as 'remote' and they are intended to receive data from all the producers We are trying to guarantee that when a local node is disconnected from network, it can still receive locally from it's local producer, and that on network recovery everything works again (all producers are visible from remote consumers, and every local consumer can receive from it's producer).
Our problem is that when we disconnect our local node (consumer + producer) from network, it stops receiving data, and it kind of hangs trying to choose a leader or something.
What would be a feasible configuration? (sorry for nonsenses cause I'm pretty new to kafka)
we configured all kafka + kraft nodes as controller and trying to get votes from all other nodes. We expected it to keeps receiving data on local disconnection from remote network but still keeping local working.
@OneCricketeer For clarifying, I mean producers as they feed data to the broker, and consumers get data from it. Message data identify source producer. I'm talking about a network split or some node breakdown... So I want to quarantee that "local" consumer can get data from "local" producer though I know it can not get data from others, in case local kafka node and local network is still working. I guess finally we are going to use a JGroups for connecting local consumer to local producer, and every producer will feed data to it's local JGroups, and also to Kafka cluster. Local consumers (they have a producer 'nearby', the only one they monitor) will listen to JGroups, and remote consumers (don't have 'nearby' producers, they listen to all the producers) will just listen to kafka cluster. Does it have sense?
