0

I will give an abstract example of the issue that I encountered.

The user makes an HTTPS request to our server (request-proxy / load balancer), the load balancer establishes a socket connection with one of the endpoint node (multi node service). This service, in turn, performs some logic, creates a message and sends it to the topic (request topic). Payload in this message also contains an assigned partitions for this instance (e.g. [1, 3, 5]). Then the system (black box) processes this request and replies to another topic (response topic) determining which partition to send this message to (e.g. randomly from [1, 3, 5]). Endpoint service (pod) that has a connections to the user receives this message and replies to user via http.

Now imagine that there was a rebalance, but the endpoint service managed to send a message before that. As a result (possibly) another pod of endpoint service will receive response but will not be able to respond to user, because no connection with him.

Note:

  1. Consumer group segregation is not the way to go (use different consumer groups for each pod), because messages are relatively large. I don't want each pod to receive messages that do not belong to it, thereby increasing the load on the network.
  2. I see no point in using key partitioning (calculate hash).

I use workarounds to solve this problem, but would really like to know what practices exist when using Kafka. Thanks.

See diagram here

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
serj026
  • 1
  • 1
  • 1
    Not sure I understand the problem. Your Kafka producer client would hash a record key the same way, regardless of what service behind the load balancer is reached – OneCricketeer Oct 05 '22 at 18:30
  • @OneCricketeer I thought about the key but I don't understand how to use it correctly in this case. Endpoint service can scale from 10 to 50 replicas (50 partitions total). The kafka request message goes through many other microservices that complement its payload. Finally, one of the microservices in the system returns a response through a response topic to particular Endpoint pod. So, what should be it a part of the message key, Endpoint pod host, pod id? – serj026 Oct 05 '22 at 19:04
  • 1
    This hash should be calculated and sent to the request topic (message payload or header), the service that generates the response must send a message to the response topic, passing this hash to key extracting it from the request message. If I understood you correctly. Kubernetes always creates / terminates pods, it's not at all clear how to calculate a unique hash for the key in this case. – serj026 Oct 05 '22 at 19:04
  • Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking. – Community Oct 06 '22 at 01:06
  • Kubernetes or replicas shouldn't matter. Your request topic can use whatever Kafka payload it wants, and the key of the Kafka record will consistently go to the same partition, unless you're using some unique pod/request ID as the record key. Regarding response topics, perhaps you should be using a stateful database to store responses since Kafka isn't really intended for request-reply use cases. The original http response can return 202 Accepted with some tracking ID, but then can poll some other GET /id endpoint for eventually consistent updates – OneCricketeer Oct 06 '22 at 01:29
  • Sorry for confusing - the question is about delivering a "response" message to specific service instance. 202 code unfortunately not suitable, the answer should be within one request. So, I got it, need to use smth else, not kafka. Current workaround - save kafka message payload to Redis if instance does not serve request (by request id), then notify other pods that response with some id saved to redis. Then load it from there and not wait for from kafka. Perhaps, ksqldb of a similar db would be suitable for this. – serj026 Oct 06 '22 at 06:02

0 Answers0