Questions tagged [kafka-partition]

For questions specifically related to Topic Partitioning in Kafka. The data within kafka topics are divided into multiple partitions. Partitioning helps in consuming faster and more parallelization can be achieved by splitting up the partitions across multiple brokers and multiple machines/servers. When using this tag also include the [kafka-topic] tag and the more generic [apache-kafka] tag where possible.

The data within kafka topics are divided into multiple partitions. Partitioning helps in consuming faster and more parallelization can be achieved by splitting up the partitions across multiple brokers and multiple machines/servers. This allows multiple consumers to consume from multiple partitions and this improves the performance of data consuming application.

52 questions
0
votes
1 answer

Decrease topic replication factor after Kafka brokers removed from cluster and failed reassignments

The topic replication factor has increased to 45 while the number of available Kafka brokers in the cluster is 40. This happened due to repeated stuck partition reassignments which were stopped. kafka-topics --topic top --zookeeper zoo_url…
rok
  • 9,403
  • 17
  • 70
  • 126
0
votes
1 answer

Total size of files in Kafka logs directory is less than the sum of their sizes

I'm testing a Kafka producer application and noticed something strange about the disk usage of the Kafka logs. When looking at the total size of a certain partition's log directory, while the application is writing to Kafka, I see this: $ ls -l…
Klitos Kyriacou
  • 10,634
  • 2
  • 38
  • 70
0
votes
1 answer

why Kafka consumer group is doing a partition revoking for different topics?

I have a consumer C1 in consumer group G1 which is reading from topic T1 (doing a poll) and in between this poll, there joins another consumer C2 in same group G1 but subscribes to a different topic T2. What I have observed is, the partition of…
0
votes
1 answer

Do Kafka's parallelism concepts apply when running in console mode?

So, I am new to Kafka and I have been reading about it for a while. I found this information on confluent. https://docs.confluent.io/current/streams/architecture.html So what I understood from this is, say I have a topic called plain_text where I…
0
votes
1 answer

Does kafka's partitions replicas should be stored with the same partition name?

Saw the picture describing kafka's partition replicas: I guess there should be the same partition (e.g. Partition 0) name for all replicas and leader, am I right?
J.J. Beam
  • 2,612
  • 2
  • 26
  • 55
0
votes
2 answers

Does Kafka supports different consumers reads the same partition with different offsets?

Does Kafka supports different consumers reads the same partition with different offsets? A partition: |------------Consumer 2, offset 2 +---+---+---+---+----+ | 0 | 1 | 2 | 3 | .. | <---- a particular…
J.J. Beam
  • 2,612
  • 2
  • 26
  • 55
0
votes
1 answer

Explain why metricbeat Kafka partition metric has a higher count than consumer metric

The problem Hi, I am trying to visualize Kafka lags using Grafana. I have been trying to log kafka lags using Metricbeat and doing the math myself since Metricbeat does not support logging Kafka lags in the version that I am using (but it has been…
Lumo Woong
  • 77
  • 1
  • 11
1 2 3
4