We are building a real time collaborative application using Redis pub-sub and Kafka. We want to move all the data received in the Redis pub-sub(irrespective of the channel) to Kafka and we came across the Redis Connector (Source and Sink) by Redis . We will be using this as a source connector. In the official docs we came across the below point.
The keys source connector does not guarantee data consistency because it relies on Redis keyspace notifications which have no delivery guarantees. It is possible for some notifications to be missed, for example in case of network failures. Docs
If the network connection between redis pubsub and the kafka redis source connector goes down then, in that period if any event occurs, the other users will receive event but the event will not be saved in the DB.
Eg: The connection between redis pubsub and kafka redis source connector is lost and at this moment user-1 changes the "name" field from "aaaa" to "asdf". The change will be affected to user-2 through the redis pubsub but in DB it will still be "aaaa".
I checked the SLA and it was at least 99.9% for both MSK Serverless and MSK Connect MSK SLA Docs. But I guess SLA has nothing to do with the network.
All the user generated events are critical for us.
Doubt #1
Is there a possibility for the connection between Redis pub-sub and Kafka Redis source connector to get lost when both the Redis pub-sub and Kafka Redis source connector are distributed across AZs ? If "yes" then how frequently the network will be lost?
Doubt #2
Instead of using a connector to take data from Redis pub-sub and put it into Kafka can the data be written from my instances(instance-1 and instance-2) to Kafka. So now my instances will make two calls. First call to produce the event to Kafka and the second call to publish the event to Redis.
Is this a scalable architecture? (I personally feel NO)
