Questions tagged [kafka-topic]
202 questions
1
vote
1 answer
kafka custom partitioner and repartitioning
We can specify custom partitioner for kafka topics. So the kafka producer can deterministically send message to a particular partition based on certain custom algorithm.
Now the question is, when I increase the number of partitions then How will…
Durgesh O Mishra
- 51
- 1
- 2
- 8
1
vote
0 answers
Speed up deletion of older records with same key (log compacted topic)
I am running Kafka broker (version 2.12) with the default server setting on my Mac. I created a log compacted topic, with the following configuration -
Configs:…
Ahmed A
- 3,362
- 7
- 39
- 57
1
vote
1 answer
parallel writes different topics from single stream topic
I have a stream which gives messages map to two different map() call and further is filtered and written to two different topics.
KStream[] stream = builder.stream("source-topic");
stream.map(logic1OnData).filter(
…
Sudarshan sridhar
- 201
- 5
- 18
1
vote
2 answers
How does Zookeeper retrive the consumer offsets from __consumer_offsets topic?
This is a followup question to "Where do zookeeper store Kafka cluster and related information?" based on the answer provided by Armando Ballaci.
Now it's clear that consumer offsets are stored in the Kafka cluster in a special topic called…
samshers
- 1
- 6
- 37
- 84
1
vote
1 answer
How to separate topic partitions into multiple kafka servers?
I want to separate topic partitions into multiple kafka servers like on this picture.
Example: I have only one topic "Test", I have two kafka servers (k1, k2), I have 4 partitions of topic "Test". I want to server k1 have partitions 0 and 1, and…
biunovich
- 13
- 4
1
vote
2 answers
Kafka Producer causes org.apache.kafka.common.network.InvalidReceiveException: Invalid receive
I am running 3-zookeeper-cluster and 3-kafka-cluster on Kubernetes.
Kafka seems to be running.
However if I produce some message to a topic and check the topic, there's no message at all.
Here's my broker saying. That says some invalid receive or…
Jaeho Lee
- 463
- 1
- 5
- 14
1
vote
1 answer
Apache Kafka Metrics Using Spring Boot
I am new to Apache Kafka, I am trying to understand how can I monitor kafka brokers, consumers and producers and topics. I am looking for metrics and how can I expose Rest apis to get all these metrics as I have to display them on the UI. And how…
learner
- 11
- 1
1
vote
2 answers
How to view topic properties in Apache Kafka
i want to view topic level properties something like
"message.timestamp.type": "LogAppendTime",
"cleanup.policy":"compact"
is it possible to view what all properties is set at topic level?
is there any command where i can view my topic level…
Mohit Singh
- 401
- 1
- 10
- 30
1
vote
2 answers
Does the same Kafka partitions of the same topic on different brokers contains the same messages ( duplicate each other)?
Broker 1:
+-------------------+
| Topic 1 |
| Partition 0 |
| |
| |
| Topic 2 |
| Partition 1 |
+-------------------+
Broker 2:
+-------------------+
| Topic 1 |
| …
J.J. Beam
- 2,612
- 2
- 26
- 55
1
vote
0 answers
how to set up ACL on a kafka topic using java api?
I am trying to set up acl on Apache Kafka topics through Java. I came across the setAcl() method present in Kafka admin client. But not sure how to use it. I have a Kafka cluster set up with 3 brokers and acl is applied on it ( server side ).
I need…
ishita07
- 37
- 7
1
vote
2 answers
How is data being added into partitions in Apache Kafka?
Hi I am working on kafka. I am trying to understand basics of kafka. I am learning kafka now. I installed kafka using docker. Currently I have one broker. I created topic with 3 partitions using below command.
kafka-topics --create --zookeeper…
Niranjan
- 1,881
- 6
- 44
- 71
1
vote
0 answers
Unable to delete Kafka topic using AdminClient deleteTopic
I am using Kafka_2.12 version 2.3.0. For topic delete purpose I am using the below code snippet
AdminClient adminClient = KafkaAdminClient.create(conf);
adminClient.deleteTopics(Collections.singleton(topic));
But when the above code is being…
Abhishek Gharai
- 227
- 3
- 15
1
vote
2 answers
How to create a Kafka consumer listener with a spring start that, in case of rejection of the message, retry to consume them after a variable time
I have a simple kafka consumer listener in a springboot application, like this:
@KafkaListener(topics="mytopic")
public void receive(String message) {
LOGGER.info("received message='{}'", messge);
}
in some particular cases I would like to…
multiplatform
- 853
- 6
- 6
1
vote
1 answer
How to read Kafka Topic line by line in Flink program
First, I load a CSV file in Kafka topic and I can print the Topic via Flink program in. The code is in following:
final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
Properties prop = new Properties();
…
M_Gh
- 1,046
- 4
- 17
- 43
1
vote
1 answer
Kafka messages not getting purged
I am new to Kafka. I am doing some experiment as to how to purge messages in a kafka topic. I found that if we set "retention.ms" property for a topic to some less time value lets say 1 second, then after 1 seconds the messages in the topic will be…
Rahul Vedpathak
- 1,346
- 3
- 16
- 30