Stack Overflow
Stack Overflow

Questions tagged [rebalancing]

47 questions
2
votes
1 answer

Kafka rebalancing in case when one subscriber in group has been strictly assigned to specific partition

Kafka rebalancing is designed to redistribute all the partitions of topic to subscriber group alive members so that any of the topic partitions is used by only one consumer at any given moment. So in case if consumers have just subscribed to a…
Suren Aznauryan
  • 984
  • 10
  • 24
1
vote
1 answer

Check the values of max.poll.interval.ms and max.poll.records

In order to prevent Kafka rebalancing, I want to increase max.poll.interval.ms and decrease max.poll.records. Also, I understand that by default is configured to: max.poll.records = 500 max.poll.interval.ms = 300000 What I am not sure about is how…
Tom Carmi
  • 385
  • 1
  • 5
  • 18
1
vote
1 answer

Rebalancing algorithm with restrictions

Please help in solving the following problem. The following entities are given: Application. Applications reside on storage and they generate traffic through service node. Service. Service is divided into several nodes. Each node has access to…
alexlys
  • 49
  • 5
1
vote
2 answers

Partitions processing stuck until state store is rebuilt during rebalancing in Kafka Streams

Let's assume I have stateful Kafka Streams application that consumes data from topic with 3 partitions. At the moment I have 2 instances of the above application running. Let's put it like that: instance1 have partitions part1 and part2 assigned,…
Nementaarion
  • 222
  • 2
  • 13
1
vote
0 answers

Rebalance in progress exception occurs unpredicatably

When running a project I encountered some RebalanceInProgressException which occurrs once or twice a week and make the consumer fail. Here is the message: org.apache.kafka.common.errors.RebalanceInProgressException: Offset commit cannot be…
Some Name
  • 8,555
  • 5
  • 27
  • 77
1
vote
0 answers

Kafka Streams exactly-once re-balance aggregation state data loss

Running 3 Kafka Streams instances with exactly-once, but experiencing loss of data when restarting one of the streams instances (the other 2 doing re-balance). If I restart the instance quickly (within session.timeout.ms), without the other 2 doing…
erikdjerf
  • 11
  • 1
1
vote
2 answers

MarkLogic: How to speed up rebalancing process when adding a new forest to an existing database?

Our production MarkLogic DB is having 1.2 TB data divided among 6 forests. We plan to add 2 new forests to reduce stands per forest count. Now, adding new forests starts rebalancing the data. That's okay, it takes time. But this rebalancing time…
akash kapil
  • 21
  • 2
1
vote
2 answers

Why can't a new-added HBase region server get regions assigned to?

When I added a new region server to the HBase cluster，no regions were assigned to this new region server. The new region server is now present on the web UI but its Num. Regions and Requests Per Second are both ZERO according the web UI. This is…
A.Chinese
  • 133
  • 11
1
vote
1 answer

Apache Ignite 2.4 uneven partitioning of data causing nodes to run out of memory and crash

Environment: Apache Ignite 2.4 running on Amazon Linux. VM is 16CPUs/122GB ram. There is plenty of room there. 5 nodes, 12GB each cacheMode = PARTITIONED backups = 0 OnheapCacheEnabled = true atomicityMode = ATOMIC rebalacneMode =…
Al A
  • 175
  • 2
  • 15
1
vote
1 answer

couchbase Re-balance failing with error - Rebalance exited with reason {badmatch,failed}

I am setting up a cluster. I tried to join 3 nodes but while re-balancing. I got below error. So i extracted some info from debug.log and unable to identify the exact issue. Appreciate any help. =========================CRASH…
Mari U
  • 21
  • 2
0
votes
0 answers

Balancer is not available

command rebalance is not currently available for execution because role balancer must be in state N/A, but it is in state Busy I cant start rebalance on cdh 5.16.2.  I search process rebalance ps -ef | grep balancer I cant resolve it (
Дмитрий Балак
  • 1
  • 1
0
votes
0 answers

Kafka connect constant rebalancing

I am deploying a kafka connect cluster, consisting of 4 workers using docker swarm. There are some cases at the initial deployment (when no other kafka connect cluster ever existed within the environment) and only then so far, that the workers…
Alexandros Mavrommatis
  • 31
  • 8
0
votes
0 answers

Is it supported to use CooperativeStickyAssignor for kafka cluster version 2.2.1?

Kafka broker cluster version is 2.2.1. Client version is 2.7.1. There is CooperativeStickyAssignor in kafka client lib. Is it supported to use CooperativeStickyAssignor to consume from a 2.2.1 broker cluster?
DeepNightTwo
  • 4,809
  • 8
  • 46
  • 60
0
votes
0 answers

Kafka consumer replicas missing messages during rolling update deployment

We have 3 spring kafka consumers, using ackmode.batch (from my understanding, it will only commit the offsets after it processed the whole batch). As well as 3 partitions with all a committed offset. During rolling update re-deployment, we are…
Erik W
  • 63
  • 1
  • 7
0
votes
1 answer

python Apache Kafka CommitFailedError due to group rebalanced

I have several kafka consumers receive message from the same topic. The message indicates which consumer is corresponding for processing that message. They all commit the message once they receive the message, and only process the message if the id…
dolker
  • 3
  • 2
1
2
3 4