I'm playing around with Kafka topics and rest-proxy and faced a behavior I don't understand.
I have 1 zookeeper, 1 kafka broker, 1 schema registry and 1 rest-proxy instance. Then the topic with 1 partition and 2 consumers reading from it are created.
I created a topic with the following command:
kafka-topics --create --if-not-exists --zookeeper localhost:32181 --partitions 4 --replication-factor 1 --topic my-ttopic
Then I tried to read from both of the consumers. Upon the second consumer is created and attempt to read using it is made some rebalancing is triggered and reading just hangs. The Kafka Rest in turns produces tons of logs like this:
[2020-10-11 09:29:30,645] INFO [Consumer clientId=consumer-grps-2, groupId=grps] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2020-10-11 09:29:30,649] INFO [Consumer clientId=consumer-grps-2, groupId=grps] Join group failed with org.apache.kafka.common.errors.MemberIdRequiredException: The group member needs to have a valid member id before actually entering a consumer group. (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2020-10-11 09:29:30,649] INFO [Consumer clientId=consumer-grps-2, groupId=grps] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2020-10-11 09:29:30,650] INFO [GroupCoordinator 2]: Preparing to rebalance group grps in state PreparingRebalance with old generation 1 (__consumer_offsets-14) (reason: Adding new member consumer-grps-2-0b19385b-df80-4d11-8eaa-7c313354a131 with group instance id None) (kafka.coordinator.group.GroupCoordinator)
[2020-10-10 20:44:51,713] INFO [Consumer clientId=consumer-grps-2, groupId=grps] Attempt to heartbeat failed since group is rebalancing (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2020-10-10 20:44:54,726] INFO [Consumer clientId=consumer-grps-2, groupId=grps] Attempt to heartbeat failed since group is rebalancing (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2020-10-10 20:44:57,739] INFO [Consumer clientId=consumer-grps-2, groupId=grps] Attempt to heartbeat failed since group is rebalancing (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2020-10-10 20:45:00,752] INFO [Consumer clientId=consumer-grps-2, groupId=grps] Attempt to heartbeat failed since group is rebalancing (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2020-10-10 20:45:03,766] INFO [Consumer clientId=consumer-grps-2, groupId=grps] Attempt to heartbeat failed since group is rebalancing (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2020-10-10 20:45:06,780] INFO [Consumer clientId=consumer-grps-2, groupId=grps] Attempt to heartbeat failed since group is rebalancing (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2020-10-10 20:45:09,793] INFO [Consumer clientId=consumer-grps-2, groupId=grps] Attempt to heartbeat failed since group is rebalancing (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2020-10-10 20:45:12,806] INFO [Consumer clientId=consumer-grps-2, groupId=grps] Attempt to heartbeat failed since group is rebalancing (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2020-10-10 20:45:15,818] INFO [Consumer clientId=consumer-grps-2, groupId=grps] Attempt to heartbeat failed since group is rebalancing (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2020-10-10 20:45:18,829] INFO [Consumer clientId=consumer-grps-2, groupId=grps] Attempt to heartbeat failed since group is rebalancing (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2020-10-10 20:45:21,841] INFO [Consumer clientId=consumer-grps-2, groupId=grps] Attempt to heartbeat failed since group is rebalancing (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2020-10-10 20:45:24,854] INFO [Consumer clientId=consumer-grps-2, groupId=grps] Attempt to heartbeat failed since group is rebalancing (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2020-10-10 20:45:27,867] INFO [Consumer clientId=consumer-grps-2, groupId=grps] Attempt to heartbeat failed since group is rebalancing (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2020-10-10 20:45:30,880] INFO [Consumer clientId=consumer-grps-2, groupId=grps] Attempt to heartbeat failed since group is rebalancing (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2020-10-10 20:45:33,893] INFO [Consumer clientId=consumer-grps-2, groupId=grps] Attempt to heartbeat failed since group is rebalancing (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2020-10-10 20:45:36,905] INFO [Consumer clientId=consumer-grps-2, groupId=grps] Attempt to heartbeat failed since group is rebalancing (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
Can you give any advice on what the problem might be about?
