0

Suppose, I have 3 Kafka broker, a zookeeper, 50 producers, 50 consumers, and 1 topics (testTopic1). And All the consumer are subscribed to testTopic1. Now I will send 50 messages at the same time with the 50 producers to the same topic (testTopic1) . Now I want that Kafka cluster do not send more than 40 messages at the same time to consumers. The remaining 10 will keep on queue or drop it.

Maybe it is a load balancing in Kafka. I do not understand how I will do this work. Im new in Kafka please help.

Giorgos Myrianthous
  • 36,235
  • 20
  • 134
  • 156
SNA Nilim
  • 211
  • 1
  • 3
  • 8

2 Answers2

1

Kafka brokers are dumb. They cant limit/remove message published to kafka.

If all kafka consumers are part of same consumer group, and there are 50 consumers, then all consumers may or may not receive all those 50 messages at same time, depending on the key. If multiple messages have same key then all same key messages will be listened by single consumer one by one. If all 50 messages have distinct keys, then they they may or may not (depending on hash of the key) will be listened by same or different consumers.

Can you explain your use case more for better understanding.

Rajat Goel
  • 2,207
  • 17
  • 35
  • Sure... i am use this on a micro-service. each of my services are communicate with a external node. and those external node have some limitation that i can not sent 50 request at a time. that's why i want to control my request. – SNA Nilim Jul 14 '19 at 13:06
  • What is the meaning of '50 request at a time'? 50 calls per second or what? – Rajat Goel Jul 14 '19 at 13:08
  • Concurrent 50 calls. 50 calls just an example. it can be more or less. – SNA Nilim Jul 14 '19 at 13:27
  • There is nothing like concurrent. It can be 50 calls/second or 50 calls/nanosecond, or somewhat else, but nothing like 50 concurrent. It needs to measurable. – Rajat Goel Jul 14 '19 at 13:29
  • 50 calls per second. – SNA Nilim Jul 14 '19 at 13:34
  • if my consumers are less than 5 is it possible that they are receive 50 request per second.? – SNA Nilim Jul 14 '19 at 13:43
  • Each consumer will pick a message from the queue one by one. And one message will be picked by only a single consumer (if all part of the same consumer group). Say it takes 10 milliseconds to process a message (including calling an external node and receiving a response from it) then a single consumer can also make more than 50 requests per second. So it depends on process time for each message – Rajat Goel Jul 14 '19 at 13:49
0

Kafka broker cannot drop messages randomly. But you can implement logic within consumer to drop the message while processing.

If you have a single topic and single partition for that topic; one among your consumers belong to the same consumer group will process all your messages since partition guaranteed ordering in processing in consumer end.

If you have 10 consumer groups and each belongs to 5 consumers and there is a single partition for the topic, at least 10 consumers process your message from topic. In case one of the consumer from consumer-group-1 fails to process the message, another consumer from same consumer group will process the message.

If you have the requirement to drop randomly 1 out 10 messages while processing, you can achieve it by adjusting the logic in consumer end. But as per consumer group offset according to broker all data is processed in its end, if system configured to maintain offset management in brokers side.

darkDragon
  • 455
  • 3
  • 12