Stack Overflow
Stack Overflow

Questions tagged [rebalancing]

47 questions
0
votes
1 answer

Kafka duplicate consumption expected as a result of rebalancing?

This is a conceptual description of a Kafka duplicate consumption scenario that we are seeing in our load test environment: GIVEN Two application instances on separate EC2 instances with one Kafka consumer each and same consumer group ID. GIVEN…
Mikael Carlstedt
  • 24
  • 4
0
votes
0 answers

Kafka consumer rebalance issue, offsets points back to very old committed offset

We are facing an issue while the client consumer restart (again not all restarts are ending up with this issue) and during the re-balancing scenario, sometimes one of the partition offsets goes back a long way from the committed offset. Kafka…
chetan
  • 1
0
votes
0 answers

How a consumer in Kafka knows when another consumer takes ownership of a partition it currently owns during rebalanceing?

As I read Kafka's paper, I found the papers says during rebalance, "when there are multiple consumers within a group, each of them will be notified of a broker or a consumer change. However, the notification may come at slightly different times at…
fsc1118
  • 1
0
votes
0 answers

Duplicate messages when using kafka mirrormaker at the time of problems on the source cluster

We have a remote kafka cluster that belongs to an external service, with which we pull data using a mirrormaker to our internal kafka cluster. The following situation has occurred - on the side of the external service, one of the cluster brokers has…
Pablinho
  • 1
  • 2
0
votes
0 answers

Routing messages for endpoint services using Kafka

I will give an abstract example of the issue that I encountered. The user makes an HTTPS request to our server (request-proxy / load balancer), the load balancer establishes a socket connection with one of the endpoint node (multi node service).…
serj026
  • 1
  • 1
0
votes
1 answer

What happens if I run the same flink application twice?

Currently, I am using KafkaSource to get the records from multiple event streams. Each event stream is having one partition. I would like to check if rebalancing happens across the Flink app's having same consumer group. Expectation: One Flink…
Chuni Lal Kukreja
  • 11
  • 3
0
votes
0 answers

Kafka Polling and rebalancing issue

I am pushing events to topic 1 and if that gets failed, then adding that event to DLQ topic2. I am using separate consumers for both of the topics and the events which are pushing into DLQ, adding an identifier(timestamp) in that event. Now, what i…
Rajat kumar
  • 1
  • 3
0
votes
1 answer

Incompatibility between Spring Kafka consumers with versions 2.3.13.RELEASE and 2.8.3

I am having problems with kafka consumers of applications with different versions of Spring-kafka, specifically between 2.3.13.RELEASE and 2.8.3. When deploying to PRO using blue/green strategies for deployment, I am having the problem that all…
Fernando Blanch
  • 131
  • 5
0
votes
1 answer

R: create column to determine wether stock is bought, held or sold

I have a dataset of portfolio components that gives information about the 10 stocks held in my portfolio at each rebalancing date(monthly). An example dataframe with just 4 dates and 4 stocks: Date <- c(rep(as.Date("2010/12/14"), 4),…
signe
  • 37
  • 4
0
votes
1 answer

ROSE: data argument is of the wrong type in R

I am trying to execute a function like the following to balance a train set with the package ROSE: library(ROSE) rose <- function(df){ str(df) set.seed(124) intrain <- sample(seq_len(nrow(df)), size = floor(0.7 * nrow(df))) train <-…
s.dallapalma
  • 1,225
  • 1
  • 12
  • 35
0
votes
0 answers

Handle Rebalancing of consumer group on addition of consumer

I want to know how can we handle consumer group re-balancing in case of a new consumer being added to the group. As it automatically rebalances the partition assignment , the existing consumer running when try to process the batch and commit the…
user3799451
  • 1
0
votes
1 answer

Kafka Connect: java.lang.IllegalStateException: No current assignment for partition

I'm running Kafka Connect on Kubernetes (8-16 nodes, autoscaled). I've defined a total of 44 connectors, one per Kafka topic (one partition per topic). Those topic are produced by Debezium / Postgresql. There are 3 Kafka nodes. Each connector has…
Wojciech Sznapka
  • 117
  • 3
  • 14
0
votes
1 answer

How to transfer business logic related state between Apache Helix instances?

Is Apache Helix capable of rebalancing data between nodes? For examle I'm developing an in-memory database where some of the partitioned/sharded data needs to be moved to another node. Is messaging or the Application Property Store a way to solve…
Towen
  • 175
  • 1
  • 1
  • 10
0
votes
2 answers

Rebalancing portfolio with given weights

Given that I have monthly weights to use in a portfolio, but I want to rebalance my portfolio say, quarterly. How may I create a function that subtracts the weights only each quarter from my original monthly weights, and then repeat those weights…
Quant
  • 27
  • 7
0
votes
0 answers

Kafka consumer fails on commit offset and rebalancing

i have a Kafka consumer which is subscribed to only one topic. At a certain point in time, after working correctly, i get the following messages in my logs: Line 25694: 2017-05-15 17:59:53.656 [INFO ] [MeasureConsumerExecutor] AbstractCoordinator -…
vortex.alex
  • 1,105
  • 3
  • 11
  • 24
1 2
3
4