Questions tagged [kafka-topic]
202 questions
1
vote
1 answer
How do I configure my topics if want to treat Kafka as a persistent event storage?
We're using Apache Kafka together with Strimzi on Kubernetes. If we want to treat Kafka as a persistent event storage (keeping events indefinitely) how would I have to configure my Topics?
Is my assumption correct, that if we leave the default…
Andreas
- 63
- 4
1
vote
2 answers
Kafka is not sending messages to other partitions
Apache Kafka installed on Mac (Intel).
Single local producer and single local consumer.
1 topic with 3 partitions and 1 replication factor is created:
bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic animal --partitions 3…
Prathamesh
- 65
- 2
- 10
1
vote
1 answer
Do we need to know number of partitions for a topic beforehand?
We want to put messages/records of a different customers on different partitions of a kafka topic.
But number of customers is not known in prior. So how can we set partition count for kafka topic in this case? Do we need any other way where…
Sai Satwik Kuppili
- 143
- 1
- 11
1
vote
0 answers
In Apache Kafka, are there any standards or common practices around how topics should be named or how topic messages should be structured?
I’m wondering if there are patterns or idioms that are common in the world of Kafka topics and messages, particularly in systems with numerous types of CRUD operations happening against numerous types of resources (e.g. users, products, orders,…
Greg Thomas
- 397
- 3
- 13
1
vote
1 answer
Create Kafka topic in Active-Passive Kafka cluster architecture
I'm trying to figure out what is the proper way to create a topic in Active-Passive Kafka cluster architecture.
Let's say we have two Kafka clusters (Active-Passive) in two separated availability zone.
I understand that the topic mirroring happens…
Denis Voloshin
- 768
- 10
- 32
1
vote
1 answer
Eviction of data from Kafka key value state store
I am performing aggregation using kafkaStreams which actually keeps all my aggregated records into a keyValue state store against a specific key which i am generating to uniquely identify that aggregation.
I am not using any kafka window for this…
pranayd
- 11
- 2
1
vote
2 answers
Kafka Topic and Partition allocation for consumer
I just started working on kafka, I need to develop a consumer client using sarama go pkg, the client is supposed to be a part of consumer group, and is needed to read from two topic A and B, client needs to read from some partitions of topic A…
Harshit Kumar
- 79
- 1
- 5
1
vote
1 answer
Are there scalability problems when having single partitions in Kafka?
If I have an application that publishes events on a kafka topic and my consumers need to read the data in the order they were published, then my topic can have only one partition, since kafka guarantees ordering only within partitions.
However, I…
L.Gashi
- 183
- 1
- 11
1
vote
1 answer
Generate a key for kafka messages that haven't any, in two topics, in order to merge / join them
From some row data for input in a poids_garmin_brut topic:
Durée,Poids,Variation,IMC,Masse grasse,Masse musculaire squelettique,Masse osseuse,Masse hydrique,
" 14 Fév. 2022",
06:37,72.1 kg,0.3 kg,22.8,26.3 %,29.7 kg,3.5 kg,53.8 %,
" 13 Fév.…
Marc Le Bihan
- 2,308
- 2
- 23
- 41
1
vote
2 answers
The Kafka topic is here, a Java consumer program finds it, but lists none of its content, while a kafka-console-consumer is able to
It's my first Kafka program.
From a kafka_2.13-3.1.0 instance, I created a Kafka topic poids_garmin_brut and filled it with this csv:
kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic…
Marc Le Bihan
- 2,308
- 2
- 23
- 41
1
vote
1 answer
Correct Number of Partitions/Replicas for @RetryableTopic Retry Topics
Hello Stack Overflow community and anyone familiar with spring-kafka!
I am currently working on a project which leverages the @RetryableTopic feature from spring-kafka in order to reattempt the delivery of failed messages. The listener annotated…
Tucker Mullens
- 33
- 2
1
vote
1 answer
How to get Load of my Kafka Topic. Is there any API with which I can measure the load on my Kafka Topic?
I have a Kafka topic on my spring boot application on which I am sending some data from producer. I want to check the load of my topic so that I can create new topics, if load on the previous topic exceeds
Fardeen mirza
- 75
- 1
- 6
1
vote
1 answer
Why doesn't a KafkaConsumer throw an error when a non-existing topic is assigned?
I have written a KafkaConsumer. The configuration looks like this:
@Bean
Map consumerConfig(
@Value("${spring.kafka.bootstrap-servers}") String bootstrapServers) {
return Map.of(
…
Harold L. Brown
- 8,423
- 11
- 57
- 109
1
vote
1 answer
Topic messages disappears from confluent topics after refreshing page
Topic messages are disappearing from a topic when using confluent client. The only ones I can see (while not reloading page), are messages which I create using the "Produce" option in the same page. Kafka configurations are ok (I think), but I still…
cristhian gallo
- 11
- 3
1
vote
2 answers
How to assign specific ID to Kafka Topic Partition
I am new to Apache Kafka. I want to assign a our user id as id to the topic partition. Is there a way to assign our own user-id to partition. I did research for couple hours on this, but didn't find any article related to assigning an ID to…
Awesome
- 5,689
- 8
- 33
- 58