Questions tagged [kafka-topic]
202 questions
0
votes
2 answers
Kafka - How to move expired messages to different topic?
The requirement is to have a Kafka topic accept messages. But the message has to be consumed and acknowledged within a certain time. If not another process should pick up the message and start a different process (say abort or rollback process).
One…
Jawahar
- 4,775
- 1
- 24
- 47
0
votes
0 answers
Writing kafka topic data into s3 bucket as csv using kafka connector
I am using Kafka connector to write kafka topic data into s3 bucket in the form of csv. I am using the following curl command
curl -X POST \
localhost:8084/connectors \
-H 'Content-Type: application/json' \
-H 'Accept: application/json' \
…
user3279893
- 147
- 3
- 16
0
votes
0 answers
concurrent consume a Specified message in two @StreamListener
i have three difference topic that called: A - B - C.
i produce a message in the "A" topic and I want "concurrent" consume that message in both B & C consumer.
even though I set concurrency in application properties, my spring cloud application…
0
votes
1 answer
Topic partition appears as null when application is run on local Kafka instance
I have this producer:
public class ProducerDemo {
public static void main(String[] args) {
String inputFile = "C:\\Users\\path\\to\\binary\\file";
String bootstrapServers = "127.0.0.1:9092";
try (
…
runnerpaul
- 5,942
- 8
- 49
- 118
0
votes
0 answers
Is creating a multi entity topic better than having all entities in separate topic?
I have looked into this article and still i have some confusion regarding merging separate topics in one comprehensive topic: https://www.confluent.io/blog/put-several-event-types-kafka-topic
So i have two entities, Account & Client as given…
Darsin
- 123
- 2
- 11
0
votes
1 answer
What happens when min.compaction.lag.ms is greater than log.cleaner.max.compaction.lag.ms?
I'm using Kafka 2.2.0, so the option to configure max.compaction.lag.ms per topic is not yet available.
Globally I have log.cleaner.max.compaction.lag.ms=86400000 (one day):
For the topic __consumer_offsets, I'd like to delay the compaction to at…
Tobias Hermann
- 9,936
- 6
- 61
- 134
0
votes
1 answer
Create kafka topic using predefined config files
Is there any way to create kafka topic in kafka/zookeeper configuration files before I will run the services, so once they will start - the topics will be in place?
I have looked inside of script bin/kafka-topics.sh and found that in the end, it…
msangel
- 9,895
- 3
- 50
- 69
0
votes
1 answer
Decrease topic replication factor after Kafka brokers removed from cluster and failed reassignments
The topic replication factor has increased to 45 while the number of available Kafka brokers in the cluster is 40.
This happened due to repeated stuck partition reassignments which were stopped.
kafka-topics --topic top --zookeeper zoo_url…
rok
- 9,403
- 17
- 70
- 126
0
votes
0 answers
Where is offset of consumer stored in Kafka
In Kafka, I know that there is a topic, named __consumer_offsets, which is used to store the offsets of consumers. As my understanding, a consumer will do two kinds of things:
fetch data from partition.
write the offsets to the topic…
Yves
- 11,597
- 17
- 83
- 180
0
votes
1 answer
How to access Kafka through nodeport
I want to access my Kafka cluster using nodeport.Here is my CRD using which I am trying to expose Kafka using nodeport.
apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
name: my-cluster
spec:
kafka:
version: 2.6.0
replicas: 3
…
rohit290554
- 79
- 3
- 10
0
votes
1 answer
Want some practical example how to use kafkaUser
I am using Kafka with strimzi operator. I don't know how to use KafkaUser can anyone please suggest to me where I should learn it's practical implementation. I just created a Kafka user and KafkaTopic now I am totally blank about what to do.
This is…
rohit290554
- 79
- 3
- 10
0
votes
0 answers
after kafkaTemplate.send(topic, avdlObject); appends dataType in value
I have kafka template and avdl schema.
avdl schema format:
recors name{
union {null, int } a;
union {null, string} b;
}
now i call kafkaTemplate.send(topic, nameObj);
when i try to consume this usig consfluent 5.4 then it return data in…
Himanshu Goel
- 91
- 6
0
votes
1 answer
What happens to data in a Kafka topic when retention.bytes limit is reached and cleanup.policy is set to compact?
What happens to data in a Kafka topic when retention.bytes limit is reached and cleanup.policy is set to compact? Which setting takes precedence? For example, if I create a topic by setting retention.bytes to 10000000 (10 MB) and cleanup.policy to…
Shiva Prathipati
- 26
- 3
0
votes
1 answer
View messages on Kafka topic that is listed on a Kamon dashboard
I can see a Kafka topic on a Kamon dashboard. How do I now view what messages are on the topic?
runnerpaul
- 5,942
- 8
- 49
- 118
0
votes
1 answer
Inputing data in a Kakfa Topic from XML with FilePulse - ExplodeFilter
I'm creating a kakfa topic which comes from an xml and writes to the topic in avro format.
I'm using the file pulse to do this, and in the documentation I saw the ExplodeFilter.
I tried to configure according to the documentation, but it is not…
Victor
- 47
- 8