Questions tagged [kafka-topic]
202 questions
2
votes
1 answer
Size of Kafka Partitions and writing to a Topic with no free disk space
I am starting to learn Kafka for enterprise solution purposes.
During my readings, some questions came to my mind:
Are all the partitions on a topic of the same size?
What will happen if a Kafka message is sent on a topic and the topic is…
Saurabh Prakash
- 2,715
- 1
- 11
- 17
2
votes
2 answers
Kafka topics with different retention although having the same configuration
I have a question regarding retention of topics messages.
I have the following situation:
Two Kafka topics;
Broker has properties log.retention.hours=1;
Describe command for each of the two topics:
[oracleas@zkafka01vdc bin]$ ./kafka-topics.sh…
F.Lazarescu
- 1,385
- 2
- 16
- 31
2
votes
2 answers
How to index Kafka topic key value as a field using logstash?
I would like to know to include Kafka topic key value along with the message as a separate field for indexing in Elasticsearch using Logstash .. the Kafka 'topic' weather contains the following messages as key value pairs:
BOSTON:99
BOSTON:89…
CuriP
- 83
- 10
2
votes
1 answer
Zookeeper: cnxn.saslServer is null and Kafka:the quorum member's saslToken is null
To provide access to only kafka for creating, deleting topics, I am creating a plaintext SASL security between kafka and zookeeper. I get the following error and can not figure out why.
zookeeper_1 | 2020-07-20 10:19:06,907 [myid:] - ERROR…
Imam Bux
- 1,006
- 11
- 27
2
votes
1 answer
Where does zookeeper store kafka cluster and related information?
By saying cluster info, I am referring to information like
subscribed consumers/consumer groups
read and committed offsets
leaders and followers of a partition
topics on the server
etc.
Does zookeeper keep this info in its own db (though I never…
samshers
- 1
- 6
- 37
- 84
2
votes
2 answers
Should topic partitions be replicated across all broker nodes in a Kafka cluster?
Though answers are available to question similar to above. My curiosity lies in the fact that suppose n1-5 nodes are in cluster where topics t1 is on n1,n2 and n3, topic t2 is on n3,n4,n5. Now if suppose p1 pushes messages in t1 and c1 consumes from…
dvsakgec
- 3,514
- 4
- 28
- 35
2
votes
0 answers
kafka electing new leader after 2/3 is down
2 of my 3 Kafka cluster is down with replica_factor=2ץ
I don't care about the DATA, I just want to save my Kafka and reuse it's topics.
I test one of my topic who has 4 partitions,
I added unclean.leader.election.enable=true
to the server.conf
but…
NoamiA
- 521
- 4
- 19
2
votes
1 answer
Can Kafka Streams output topic be on a separate cluster?
I have a topic where all logs are pushed to centralized topic but I would like to filter out some of those records to a separate topic and cluster if possible.
Thanks
user432024
- 4,392
- 8
- 49
- 85
2
votes
3 answers
How to delete data which already been consumed by consumer? Kafka
I am doing data replication in kafka. But, the size of kafka log file is increases very quickly. The size reaches 5 gb in a day. As a solution of this problem, ı want to delete processed data immediately. I am using delete record method in…
omerstack
- 535
- 9
- 23
2
votes
1 answer
How to delete a topic configuration via KafkaAdminClient
I want to delete a configuration (reset it to default) for a topic which was overridden before. This is possible with the provided script
$> ./kafka-configs.sh --zookeeper localhost:2181 --alter --entity-type topics \
--entity-name test…
maltesmann
- 445
- 1
- 6
- 13
2
votes
1 answer
What is topic.registered in ksql when running list topics
I'm using kafka-connect to stream the rows from a mysql table to a kafka topic.
When I run this on my ksql-client:
LIST TOPICS;
I get a list of topics looking like this:
Kafka Topic | Registered | Partitions | Partition Replicas…
lloiacono
- 4,714
- 2
- 30
- 46
1
vote
1 answer
Joining two Java KStreams (Kafka Topics) with "nearly" identical keys
i have two kafka topics containing different information to a "Warning Event".
To know what entries in Topic A and B correspond to one another, i have to compare the serial, date and machine form the Key and then join on the 'no'
IMPORTNANT: The…
klimzera
- 11
- 2
1
vote
0 answers
Publishing an AVRO messages to topic via Kafka REST
Getting below error when producing message to topic.
"message": "Cannot construct instance of `io.confluent.kafkarest.entities.v3.ProduceRequest$ProduceRequestData`, problem: 'schema_version=17' cannot be used with 'serializer'."
I'm trying to send…
1
vote
1 answer
Create MSK Kafka topic through Terraform
I am trying to create topics through Terraform, in a AWS MSK Kafka cluster, using the Mongey/kafka provider, but I always get connection refused.
I have the cluster configured as follows:
resource "aws_msk_cluster" "this" {
cluster_name …
Homer
- 85
- 7
1
vote
2 answers
Ideal number of Kafka topics per Kafka cluster
We have a use case where we would like to have around 2000 kafka topics (we have 3 brokers) and 1 partition per topic. Do we have any known recommendation on what is the maximum number of topics that one cluster can have? I went through the…
TechEnthusiast
- 1,795
- 2
- 17
- 32