Stack Overflow
Stack Overflow

Questions tagged [kafka-consumer-api]

Use for questions related to the Apache Kafka consumer API

Use for questions related to the Apache Kafka consumer API. Any question related to consuming from the Kafka topics. Introspecting offsets. Consumers failing and restarting reliably should use this tag.

4238 questions
1
vote
1 answer

How can encrypt the kafka password in jaas configuration file

I am new to Kafka, please let me know how can we encrypt the password used in jaas configuration file. My config example. KafkaClient { org.apache.kafka.common.security.plain.PlainLoginModule required username="username" …
fouad khan
  • 11
  • 1
1
vote
0 answers

Dotnet Kafka Consumer pause and resume on schedule/on-demand

We have a Dotnet BackgroundService(Microsoft.Extensions.Hosting) running from a WebApi project. There is a Kafka Consumer(fairly new to Kafka world) running from within the ExecuteAsync method. We would like our Kafka consumer to stop/start…
AJ_NY
  • 59
  • 1
  • 8
1
vote
2 answers

kafka could not read log4j config file from URL [file:/c/kafka_2.12-2.8.0/bin/../config/tools-log4j.properties]

I started zkper and kafka. Then I tried to run consumer and got this error. the command i used: kafka-console-consumer.sh --zookeeper localhost:2181 --topic erjan --from-beginning The error: erjan@erjancomputer MINGW64…
ERJAN
  • 23,696
  • 23
  • 72
  • 146
1
vote
1 answer

Multiple Kafka configurations in a reactive Kafka project

I am working on a project that has 2 services: both reading, transforming the messages and then writing to another Kafka. The Kafka configuration for both these services are different. Here is my application.yml spring: kafka: …
perplexedDev
  • 857
  • 4
  • 17
  • 49
1
vote
1 answer

Getting error while setting up a Kafka Producer and consumer using Kafkacat

used these commands to set up a kafka server and then build a basic producer consumer model for sending hello from producer to consumer. brew services start zookeeper brew services start kafka kcat -P -b localhost:9092 -t topic1 kcat -C -b…
arush saxena
  • 57
  • 6
1
vote
1 answer

CommitFailedException by Spring Kafka Consumer

I get the below error message sometimes while using Spring Kafka Consumer .I have implemented at least once semantics as shown in the code snippet 1 )My doubt is do I miss any message from consuming? 2)Do i need to handle this error .As this error…
Pale Blue Dot
  • 511
  • 2
  • 13
  • 33
1
vote
1 answer

Can't convert value of class org.json.JSONObject to class org.apache.kafka.common.serialization.StringSerializer/ JsonSerializer

How looks my configurations: Map props = kafkaProperties.buildProducerProperties(); props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer"); props.put("value.serializer",…
Gavin Smith
  • 25
  • 4
1
vote
1 answer

@KafkaListener get all messages from particular kafka topic

I have a @KafkaListener method to get all messages in topic but I only get one message for each interval time that @Scheduled method works. How can I get all messages from topic in once? Here's my class; @Slf4j @Service public class…
Burak
  • 133
  • 2
  • 12
1
vote
1 answer

Python confluent_kafka - List All Consumer currently Listening to a Topic

How to get of consumer.id which are listening to a topic using Confluent_Kafka? I am able to get the consumer-groups and topic. But not the consumer name/id.
Marshall
  • 127
  • 3
  • 11
1
vote
2 answers

Kafka consumer and a web server simultaneously, thread blocking problem microservice

assumptions: There are microservices behind an api-gateway, they communicate through HTTP synchronously. obviously, each one of those microservices is a web server. now I want my microservice to play as a Kafka producer and "consumer" too. more…
Gholamreza Eghbali
  • 301
  • 1
  • 4
  • 10
1
vote
0 answers

Spring Kafka Ssl Implemantation without using trust/keystores

Hi I want to implement ssl feature in my spring kafka connection.To achive this I have to set SslTruststoreLocation,SslKeystoreLocations.But is it possible not to set classpath.Can I give the content o cert file as a string and running ssl…
Bilgehan
  • 1,135
  • 1
  • 14
  • 41
1
vote
3 answers

Kafka-Consumer issue, showing some time out error

I'm new to this Apache Kafka topic and I was writing some basic producer-consumer code, and I'm facing some issues with consumer code, after starting the zookeeper and Kafka, I created a topic name "firsttopic", and I'm entering some events using…
Vivek Jaiswal
  • 19
  • 7
1
vote
0 answers

Kafka is not receiving messages after adding new consumer group using kafka clients java

Kafka working fine when we push new messages but not working if we are reading old queued messages after adding new consumer group. Note : The problem is only existing with the below java configuration of Kafka else its working fine if we use spring…
Abhishek Mittal
  • 91
  • 2
  • 7
1
vote
1 answer

How to let a consumer always start reading with the last massage?

What is the best solution to let a consumer always read the last, most recent message of a key? Is there a solution in kafka or do I need kafka streams for it? An Example: The messages m with the keys K1-K3 are in a log: |K1 m0…
synpho
  • 11
  • 1
1
vote
0 answers

How to stop a consumer group which is bound to kafka cluster and then have it's offset reset/moved programatically

I have been using kafka AdminClient (below snippet) for resetting offset : Map newOffsets = getOffsetMap(); AlterConsumerGroupOffsetsResult resultst =…
Saurabh Bhoomkar
  • 595
  • 1
  • 9
  • 29
1 2 3
…
99
100