Stack Overflow
Stack Overflow

Questions tagged [kafka-topic]

202 questions
0
votes
0 answers

how to consume kafka topics in sequence in Micronaut

topic 1 and topic 2 i want to be able to consume topic 1 and only after consuming the message start consuming topic 2 message. I tried to sendTo the data from topic1 to topic2 and setting autoStartUp() property to false for topic2 but how do i hold…
Sindhu S
  • 1
  • 1
0
votes
0 answers

Creating a topic on startup of Kafka container - connection to node could not be established

I'm using the Docker compose file below to deploy a simple Kafka server in a container. I also deploy a Nodejs application that bootstraps a consumer on the topic 'my-topic'. So I need the topic to be available when I run the docker compose up…
ngAlchemist
  • 51
  • 5
0
votes
0 answers

Topic records sync is not working when topic delete from destination cluster and re-apply replication using mirrormaker2

Problem scenario we replicated topics from my source Kafka cluster to my target Kafka cluster, this of this as a merge. We have an kafka cluster that we want to move off of and the new cluster which already contains other topics. Initially, the…
govinda dangal
  • 91
  • 1
  • 3
0
votes
0 answers

Consuming time-series data from a multi-partitioned topic but maintaining order

There's a question I have about this one, but I haven't gotten a satisfactory answer yet. In time-series data, the order in which messages are sent is crucial. Let's say a downstream consumer has a Python script that computes windowed statistics on…
sci9
  • 700
  • 1
  • 7
  • 21
0
votes
1 answer

Spark consumer streaming from Kafka topic keeps resetting offset

so I'm running a spark streaming program and reading a stream from a Kafka topic with the following options for the readStream method. options = { ... ... "kafka.security.protocol": "SASL_SSL", "startingOffsets":…
user1943079
  • 511
  • 6
  • 20
0
votes
0 answers

How to deserialize the message if the data format is not known?

I am trying to consume messages from the Kafka topic but I don't know if the format of data is JSON or AVRO, so which KEY_DESERIALIZER_CLASS_CONFIG and VALUE_DESERIALIZER_CLASS_CONFIG to use. Is there any way to find out which format is used?
Harry
  • 1
  • 2
0
votes
1 answer

Kafka Spring: How to hide logs from RetryTopicConfigurer

I have a topic with two partitions,i'm using @RetryableTopic and i can see logs on my app console that INFO o.s.k.r.RetryTopicConfigurer - Received message in dlt listener: {topic name with second partition} and this is wrong because it's just…
Kamalama
  • 23
  • 2
0
votes
0 answers

Kafka KRaft - Topic Deletion

When attempting to delete a topic using the kafka-topics.sh script, I can see that the topics in the log.dir are marked for deletion, but not all partitions belonging to the topic have been marked. Eventually, the partitions marked for deletion are…
Jon Phelps
  • 1
0
votes
1 answer

Is there any way to recognize the last kafka message of every partition in multi partition topic and multiple consumers apart from lag?

Is there any way to recognize the last kafka message of every partition in multi-partitioned topic and multiple consumers apart from lag? I know one way to identify the last message which is through AdminClient API/ kafka consumer API. But need to…
Sakshi K
  • 1
  • 2
0
votes
1 answer

kafka-topics.bat --list --bootstrap-server localhost:9092 is not returning list

kafka-topics.bat --list --bootstrap-server localhost:9092 is not returning anything kafka-topics.bat --list --bootstrap-server localhost:9092 is not showing anything The above command meant to show me the topic list but it wasn't returning anything.
Harsh Kumar
  • 1
  • 3
0
votes
1 answer

MongoDB Kafka Connect can't send large kafka messages

I am trying to send json large data (more than 1 Mo ) from MongoDB with kafka connector, it's worked well for small data , but I got the following error when working with big json data : [2022-09-27 11:13:48,290] ERROR…
Zak_Stack
  • 103
  • 8
0
votes
0 answers

How to get the statistics of Kafka partition

We want to get the statistics for partition through the command or exposed API, like size of the partition , no of event in a partition for a topic.
Nitul
  • 997
  • 12
  • 35
0
votes
1 answer

Kafka permission on a topic creating a Group Authorization Exception

So I have a Kafka cluster running with zookeeper with SSL. I gave a read permission to a user for a specific topic on the Kafka ACL: I can see it in zookeeper. When this user is consuming the data, they are getting a Group Authorization…
CMPE
  • 1,853
  • 4
  • 21
  • 37
0
votes
1 answer

Kafka: "Broker failed to validate record" after increasing partition

I had increased the partition of an existing Kafka topic via terraform. The partition size had increased successfully however when I test the connection to the topic, I'm getting a "Broker failed to validate record" Testing method: echo "test" |…
Elisha Choo
  • 1
  • 2
0
votes
1 answer

Kafka topics as chatroom identifiers

I am interested in a messaging chat app where different chatroom can be possible. I try to understand if Kafka topics can be used as chatroom identifiers from Kafka conceptual point of view? As chatrooms are normally created in a runtime then it…
Mikhail Geyer
  • 881
  • 2
  • 9
  • 27
1 2 3
…
13 14