Stack Overflow
Stack Overflow

Questions tagged [kafka-partition]

For questions specifically related to Topic Partitioning in Kafka. The data within kafka topics are divided into multiple partitions. Partitioning helps in consuming faster and more parallelization can be achieved by splitting up the partitions across multiple brokers and multiple machines/servers. When using this tag also include the [kafka-topic] tag and the more generic [apache-kafka] tag where possible.

The data within kafka topics are divided into multiple partitions. Partitioning helps in consuming faster and more parallelization can be achieved by splitting up the partitions across multiple brokers and multiple machines/servers. This allows multiple consumers to consume from multiple partitions and this improves the performance of data consuming application.

52 questions
1
vote
2 answers

Kafka is not sending messages to other partitions

Apache Kafka installed on Mac (Intel). Single local producer and single local consumer. 1 topic with 3 partitions and 1 replication factor is created: bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic animal --partitions 3…
Prathamesh
  • 65
  • 2
  • 10
1
vote
1 answer

Do we need to know number of partitions for a topic beforehand?

We want to put messages/records of a different customers on different partitions of a kafka topic. But number of customers is not known in prior. So how can we set partition count for kafka topic in this case? Do we need any other way where…
Sai Satwik Kuppili
  • 143
  • 1
  • 11
1
vote
1 answer

Are there scalability problems when having single partitions in Kafka?

If I have an application that publishes events on a kafka topic and my consumers need to read the data in the order they were published, then my topic can have only one partition, since kafka guarantees ordering only within partitions. However, I…
L.Gashi
  • 183
  • 1
  • 11
1
vote
2 answers

How to assign specific ID to Kafka Topic Partition

I am new to Apache Kafka. I want to assign a our user id as id to the topic partition. Is there a way to assign our own user-id to partition. I did research for couple hours on this, but didn't find any article related to assigning an ID to…
Awesome
  • 5,689
  • 8
  • 33
  • 58
1
vote
1 answer

How to specify the Kafka topic's partition in Kafka Connect Sink

I am trying to specify a topic partition for my Kafka Connect Sink. In particular, I am using the DataStax Apache Kafka Connector. There is a good amount of documentation and resources related to specifying a topic partition for a Kafka Consumer,…
RyanQuey
  • 705
  • 1
  • 9
  • 29
1
vote
1 answer

How are messages distributed in the kafka partition?

If we have one topic with 4 partitions in Kafka. There are 4 publisher which publish message in the same topic. All publisher publish different count of message like publisher1 publishes W messages, publisher2 publishes X messages, Publisher3…
Rishabh
  • 185
  • 1
  • 13
1
vote
0 answers

Kafka custom partitioner for uniform load balancing across consumers

I have 1 topic with K partitions, however the messages are of varying size. Let's say they have weights on a scale of 1 to 10, and the consumers take equivalent time/resources to process them. How do we design a custom partitioner to spread the…
vrtx54234
  • 2,196
  • 3
  • 30
  • 53
1
vote
1 answer

What happens to the offsets that are compacted in a kafka partition?

Let us say, I have a partion-0 with 2 log segments. The first segment contains keys v1, v2 and v3 at offsets 0, 1 and 2. The second segment has keys v3, v4, v1 at offsets 3, 4 and 5. When the log compaction happens, the keys at offset 0 and 2 in…
swetha
  • 91
  • 2
  • 11
1
vote
1 answer

Does Kafka consumer reads the message from active segment in the partition?

Let us say I have a partition (partition-0) with 4 segments that are committed and are eligible for compaction. So all these segments will not have any duplicate data since the compaction is done on all the 4 segments. Now, there is an active…
swetha
  • 91
  • 2
  • 11
1
vote
1 answer

Does Kafka chose an alternative partition in the same topic if the partition I want to send message to was full of disk?

My topic has 3 partitions (in 3 different brokers). I want to send message to particular partition 1 (add partition id or specify a key in my message). Assuming the partition 1 became full of disk, does an alternative partition (partition 2 or 3 in…
Hieu Doan
  • 115
  • 1
  • 7
1
vote
1 answer

How to separate topic partitions into multiple kafka servers?

I want to separate topic partitions into multiple kafka servers like on this picture. Example: I have only one topic "Test", I have two kafka servers (k1, k2), I have 4 partitions of topic "Test". I want to server k1 have partitions 0 and 1, and…
biunovich
  • 13
  • 4
1
vote
3 answers

Number of consumers in a single Kafka comsumer-group

I want to consume 3 topics, with 4 partitions each, using a single consumer group. Should the consumer group contain 3, 4, or 12 consumers? I want to achieve optimal consumption.
avivas
  • 11
  • 2
1
vote
1 answer

How to programmatically get latest offset of a Kafka topic in Java

Here is what I am trying Collection partitions = consumer.partitionsFor(topic).stream(); And also how to indicate you've hit the end or there isn't anymore messages to consume. If the offset doesn't match the broker's end offset at…
Reddy
  • 11
  • 1
  • 4
1
vote
1 answer

Test kafka behaviour when disconnecting one broker (spring-kafka)

I'm starting in Kafka. I have one cluster with 2 broker (ids #2 and #3), and a replication factor of 2. I want to test the bahaviour of Kafka if I disconnect one broker (id #3). After the shutdown of the #3, my topic informations: Topic: CUSTOMER…
Clément V
  • 23
  • 7
1
vote
2 answers

Is a write to a Kafka topic only successful if write on each partition's replicas is successful?

Is a write to a Kafka topic only successful if the write on each partition's replicas is successful? Or is there a quorum that could be configured? Suppose you have Producer Server1 with Topic1 Partition1(Leader) Server2 with Topic1…
J.J. Beam
  • 2,612
  • 2
  • 26
  • 55
1
2
3 4