Stack Overflow
Stack Overflow

Questions tagged [kafka-partition]

For questions specifically related to Topic Partitioning in Kafka. The data within kafka topics are divided into multiple partitions. Partitioning helps in consuming faster and more parallelization can be achieved by splitting up the partitions across multiple brokers and multiple machines/servers. When using this tag also include the [kafka-topic] tag and the more generic [apache-kafka] tag where possible.

The data within kafka topics are divided into multiple partitions. Partitioning helps in consuming faster and more parallelization can be achieved by splitting up the partitions across multiple brokers and multiple machines/servers. This allows multiple consumers to consume from multiple partitions and this improves the performance of data consuming application.

52 questions
1
vote
2 answers

Does the same Kafka partitions of the same topic on different brokers contains the same messages ( duplicate each other)?

Broker 1: +-------------------+ | Topic 1 | | Partition 0 | | | | | | Topic 2 | | Partition 1 | +-------------------+ Broker 2: +-------------------+ | Topic 1 | | …
J.J. Beam
  • 2,612
  • 2
  • 26
  • 55
0
votes
0 answers

Extracting values from TopicPartition - confluent kafka

Trying to extract the tuple values from TopicPartition like topic, partition, offset etc. Could not locate a helper method. Added code snippet below from confluent_kafka import DeserializingConsumer conf = { 'bootstrap.servers':…
madmatrix
  • 205
  • 1
  • 4
  • 12
0
votes
1 answer

Kafka partition configuration for kubernetes HPA

We currently have a Kafka partition with 3 and 3 Kubernetes pods for each Kafka partition. We observed that there is a performance lag because of less number of pods. We are planning to increase the pods. We observed that lag is happening during…
Spartan
  • 3,213
  • 6
  • 26
  • 31
0
votes
1 answer

Kafka Partition Does Not Match MurmurHash2 32-bit Algorithm

I'm working on a disaster recovery feature where I need to determine the Kafka partition of a given key in order to replay messages from that partition. I've read that if a key is provided to Kafka, it will use murmur2(key) % numOfPartitions however…
markmiller
  • 160
  • 1
  • 1
  • 10
0
votes
0 answers

Kafka out of order messages and kafka down error

I was having an error cannot CREATE PARTITIONS on my service logs and I rearranged my partitions in yml file according to partition count given in KafkaDrop and restarted services. But now I'm seeing related error and after some time kafka is going…
meursault
  • 255
  • 1
  • 4
  • 13
0
votes
0 answers

How Many Partitions we can create for single topic?

How Many max number of Partitions we can create for a topic in kafka ? Is that any rule or any formula for that ?
Amit Biswas
  • 5
  • 4
0
votes
0 answers

How to get the statistics of Kafka partition

We want to get the statistics for partition through the command or exposed API, like size of the partition , no of event in a partition for a topic.
Nitul
  • 997
  • 12
  • 35
0
votes
2 answers

How to increase throughput in Kafka with a constant number of partitions and dynamic number of consumers

I have 6 partitions for a certain topic, and 4 consumers consuming from that topic. The producer produces in a round robin manner to the partitions. The 4 consumers belong in the same consumer group. I can see with some load testing that 2 of the…
thepaulbot
  • 25
  • 6
0
votes
1 answer

Use kafka partitioner as a lock

I have a spring boot application that sends messages into a Kafka topic, something like: public class Message { @JsonProperty("customer_id") @SerializedName("customer_id") private String customerId; ... } For the sake of some…
Tom Carmi
  • 385
  • 1
  • 5
  • 18
0
votes
1 answer

Is it possible to control how often a Spring Kafka Message Listener switches between its assigned partitions?

When a Spring Kafka MessageListener is consuming messages from multiple partitions, it keeps processing messages from one partition until there are no more messages and only after that it continues with the next partition. (based on my…
Viorel Vesa
  • 300
  • 1
  • 2
  • 12
0
votes
1 answer

Topic partition on basis of partition Key

I am trying to split a stream into multiple partition on the basis of a partition key but apparently its not working. The implementation is such that I have a Class lets say Metrices public class Metrice { public string MetriceType { get; set;…
user_19240589
  • 97
  • 1
  • 14
0
votes
2 answers

Kafka default partitioner behavior when number of producers more than partitions

From the kafka faq page In Kafka producer, a partition key can be specified to indicate the destination partition of the message. By default, a hashing-based partitioner is used to determine the partition id given the key So all the messages with…
Gadam
  • 2,674
  • 8
  • 37
  • 56
0
votes
1 answer

Spring boot kafka: Microservice multi instances, concurrency and partitions

I have a question about the way of publishing and reading messages in kafka for microservices arquitectures with multiple instance of the same microservices for writing and reading. My main problem here is that the microservices that publish and…
Gabriel García Garrido
  • 435
  • 2
  • 12
  • 40
0
votes
1 answer

Properly Seek and Consume Kafka Messages on Multipartition Topic

I recently found that a topic i've been using is multi-partition rather than single partition. I need to reconfigure my consumer class to handle the multiple partitions, but i'm a touch confused. I am currently using an offset group, let's call it…
StormsEdge
  • 854
  • 2
  • 10
  • 35
0
votes
2 answers

Does Kafka Streams GlobalKTable topic require the same number of partitions as KStream topic which it will be joining with?

We want to use GlobalKTable in Kafka streams application. Input topics(KTable/KStream) have N partitions and a GlobalKTable will be used as a dictionary in the stream application. Does the input topic for the GlobalKTable must have the same number…
Mikhail Dubrovin
  • 88
  • 8
1 2
3
4