Stack Overflow
Stack Overflow

Questions tagged [kafka-topic]

202 questions
6
votes
2 answers

What is the maximum replication factor for a partition of kafka topic

I hava kafka cluster having 3 brokers and a couple of topics with each having 5 partitions. Now i want to set the replication factor for the partitions. What is the maximum replication factor which i can set for a partition of kafka topic?
KayV
  • 12,987
  • 11
  • 98
  • 148
6
votes
2 answers

Kafka: disable create topic from Java

Using Kafka with the Java lib, I want to disable the automatic creation of topic (if it doesn't already exist). Some sites says I should put auto.create.topics.enable to false, but this is not recognized in Java. 15:11:56.962 [main] WARN …
Rolintocour
  • 2,934
  • 4
  • 32
  • 63
5
votes
1 answer

Can compacted Kafka topic be used as key-value database?

In many articles, I've read that compacted Kafka topics can be used as a database. However, when looking at the Kafka API, I cannot find methods that allow me to query a topic for a value based on a key. So, can a compacted Kafka topic be used as a…
xarx
  • 627
  • 1
  • 11
  • 27
5
votes
3 answers

Kafka Log Compacted Topic Duplication Values against same key not deleted

Log compacted topics are not supposed to keep duplicates against the same key. But in our case, when a new value with the same key is sent, the previous one isn't deleted. What could be the issue? val TestCompactState: KTable[String, TestCompact] =…
Danish
  • 257
  • 2
  • 12
5
votes
1 answer

What are the practical limits of Kafka regex-topics / listening to multiple topics

I am exploring different PubSub platforms and I was wondering what the limits are in Kafka for listening to multiple topics. Consider for instance this Use Case. We have trains, station entry gates, devices that all publish their telemetry.…
Patrick Savalle
  • 4,068
  • 3
  • 22
  • 24
4
votes
1 answer

Does updating broker settings (topic defaults) change already existing topics?

In kafka there is a log.retention.ms configuration property that can be set at the Broker level and it can be overridden by retention.ms on topic level. Imagine I have a topic in my Kafka cluster with default config - no explicitly set retention.…
moffeltje
  • 4,521
  • 4
  • 33
  • 57
4
votes
2 answers

Python-Kafka: Keep polling topic infinitely

I am using python-kafka to listen to a kafka topic and use that the records. I want to keep it polling infinitely without any exit. This is my code below: def test(): consumer = KafkaConsumer('abc', 'localhost:9092',…
mifol68042
  • 183
  • 1
  • 4
  • 18
4
votes
1 answer

understanding kafka log.dirs

I have Kafka cluster and the log.dirs=/data/kafka is set to the data directory in server.properties. my DATA partition is kept getting full due to these logs which take a big part of it. (talking about binary logs in topic directory like…
NoamiA
  • 521
  • 4
  • 19
4
votes
1 answer

Where to set parameters min.insync.replicas and acks in Java?

I need to set two parameters min.insync.replicas and acks. The official documentation says that the parameter min.insync.replicas is the broker's parameter. Do I understand correctly that for all topics it should be specified inside the…
Violetta
  • 509
  • 4
  • 12
4
votes
2 answers

How to create Kafka topic using Segmentio's kafka-go?

Can I get an example of creating a topic using segmentio's kafka-go? I have tried creating a topic as below: c, _ := kafka.Dial("tcp", "host:port") kt := kafka.TopicConfig{Topic: "sometopic", NumPartitions: 1, ReplicationFactor: 1} e :=…
prabhu
  • 41
  • 1
  • 2
4
votes
2 answers

What is the ideal number of partitions in kafka topic?

I am learning Kafka and trying to create a topic for my recent search application. The data being pushed to kafka topics is assumed be a high number. My kafka cluster have 3 brokers and there are already topics created for other requirements. Now…
KayV
  • 12,987
  • 11
  • 98
  • 148
4
votes
3 answers

Kafka topic partitions with leader -1

I noticed that few of my kafka topics are behaving in a manner i cannot explain clearly. For eg: ./kafka-topics.sh --describe --zookeeper ${ip}:2181 --topic test Topic:test PartitionCount:3 ReplicationFactor:1 Configs:retention.ms=1209600000 …
irrelevantUser
  • 1,172
  • 18
  • 35
4
votes
2 answers

fixing kafka cluster with under-replicated partitions

We are having a problem with one of our kafka clusters. We have 6 nodes with v1.0, all the topics have a replication factor of 3 and 10 partitions/topic which seemed to be enough for us. Due to a power failure, 3 of the nodes went down for a while,…
Acampoh
  • 589
  • 1
  • 8
  • 23
3
votes
1 answer

Kafka replication factor

When we say a topic is created with a RF = 3. Does this mean each partition will have total of 3 replicas - out of which 1 will be leader and 2 will be replicas? Or does this mean that there will be 3 replicas on top of 1 leader replica and…
Ankit Sahay
  • 1,710
  • 8
  • 14
3
votes
2 answers

Define Kafka ACL to limit topic creation

We are currently running an unsecured Kafka setup on AWS MSK (so I don't have access to most config files directly and need to use the kafka-cli) and are looking into ways to add protection. Setting up TLS & SASL is easy, though as our Kafka cluster…
Dennis van de Hoef - Xiotin
  • 1,481
  • 1
  • 13
  • 29
1
2
3
…
13 14