Stack Overflow
Stack Overflow

Questions tagged [kafka-topic]

202 questions
3
votes
1 answer

Ideal number of partitions for Kafka topic

I am currently working on a setup which has 6 kafka-brokers, Data is being pushed into my topic from two producers at a rate of about 4000 messages per second, I have 5 Consumers for this topic working as a group. What should be the ideal number of…
SRI HARSHA S V S
  • 91
  • 2
  • 8
3
votes
1 answer

Kafka: auto create topic with options

I'm using auto-creation of topics in Kafka. Some topics require compaction, some not. Is it possible to specify custom options when auto-creating a topic; or do I need to test for existence and create it 'manually' with the desired options before I…
BenoitParis
  • 3,166
  • 4
  • 29
  • 56
3
votes
1 answer

Does kafka support millions of partitions?

Will we have any problem if we have millions of partitions for one topic? Due to our business requirement, we are thinking if we can make a partition for every user in kafka. We have millions of users. Any insight would be appreciated!
Moon
  • 361
  • 2
  • 10
3
votes
1 answer

Read Kafka Topic from beginning

I have a spring boot application, that keeps reading all messages from a specific topic. My problem is that I have an issue that when the application restarts, it needs to read all the messages from the topic again. I thought these two options…
Victor
  • 47
  • 8
3
votes
1 answer

Apache Flink - Partitioning the stream equally as the input Kafka topic

I would like to implement in Apache Flink the following scenario: Given a Kafka topic having 4 partitions, I would like to process the intra-partition data independently in Flink using different logics, depending on the event's type. In particular,…
Maurizio Cimino
  • 105
  • 1
  • 9
3
votes
1 answer

How to programmatically get latest offset per Kafka topic partition in Python

I'm new to Kafka, and want to get the position of a Kafka topic, per partition. I see in the documentation - https://kafka-python.readthedocs.io/en/master/apidoc/KafkaAdminClient.html#kafkaadminclient - that the offset is available via the function…
Chedva
  • 129
  • 2
  • 11
3
votes
1 answer

How to programatically create topic and send message to Kafka using C# dot net client

I'm new to kafka and I want to try create topic and send message to kafka from my .net application. I'm using kafka.net dll and created topic successfully with this code: Uri uri = new Uri("http://localhost:9092"); string topic =…
maya
  • 31
  • 1
  • 2
3
votes
1 answer

Should I create more topics or more partitions?

Kafka gets orders from others countries. I need to group these orders by countries. Should I create more topics with country name or about to have one topic with different partitions? Another was it to have one topic and use strean Kafka that…
user13419533
3
votes
1 answer

Kafka topic cannot be deleted or recreated

I have a Kafka topic that seems to simultaneously exist and not exist. kafka-topics.sh --bootstrap-server localhost:9092 --topic my-topic --delete returns an error: Topics in [] does not exist Meanwhile, trying to re-create the topic…
wrschneider
  • 17,913
  • 16
  • 96
  • 176
3
votes
2 answers

Why can it take so long to delete Kafka topics?

On a 3 node cluster, I created few topics with thousands of messages. I have noticed that it takes long time to delete a topic. It took me more than 14 mins to delete 500 topics. Are there any best practices for topic deletion? Is there any…
JR ibkr
  • 869
  • 7
  • 24
3
votes
3 answers

Why is kafka not creating a topic? - is not a recognized option

I am new to Kafka and trying to create a new topic on my local machine. I am following this https://medium.com/@maftabali2k13/setting-up-a-kafka-cluster-on-ec2-1b37144cb4e Start zookeeper bin/zookeeper-server-start.sh -daemon…
Stan11
  • 274
  • 3
  • 11
3
votes
2 answers

Splitting Kafka into separate topic or single topic/multiple partitions

As usual, it's bit confusing to see benefits of splitting methods over others. I can't see the difference/Pros-Cons between having Topic1 -> P0 and Topic 2 -> P0 over Topic 1 -> P0, P1 and a consumer pull from 2 topics or single topic/2…
Ahmed Alaa El-Din
  • 1,813
  • 1
  • 16
  • 19
3
votes
1 answer

How to programmatically create topics using kafka-python?

I am getting started with Kafka and fairly new to Python. I am using this library named kafka-python to communicate with my Kafka broker. Now I need to dynamically create a topic from my code, from the docs what I see is I can call create_topics()…
Paras
  • 3,191
  • 6
  • 41
  • 77
2
votes
1 answer

Kafka: What happens to message with null key when log compaction is enabled?

What happens to message with null key when log compaction is enabled? Are messages with a null key compacted?
Cr4zyTun4
  • 625
  • 7
  • 18
2
votes
2 answers

How to list Kafka topics while I am using strimzi operator

Hey I am usin Kafka Strimzi. I created my kafkaTopic and KafkaUser using following yml files: apiVersion: kafka.strimzi.io/v1beta1 kind: KafkaUser metadata: name: my-user labels: strimzi.io/cluster: my-cluster spec: authentication: …
rohit290554
  • 79
  • 3
  • 10
1 2
3
…
13 14