Stack Overflow
Stack Overflow

Questions tagged [kafka-consumer-api]

Use for questions related to the Apache Kafka consumer API

Use for questions related to the Apache Kafka consumer API. Any question related to consuming from the Kafka topics. Introspecting offsets. Consumers failing and restarting reliably should use this tag.

4238 questions
1
vote
1 answer

How to make sure multiple Kafka topologies have same partition assignment of all topics?

I have an use case where we have multiple Kafka topologies defined in same application. I am running multiple instances of my application (e.g. 2 topologies running on 2 instances). Let's say 1st topology have 2 topics (t0, t1) with 2 partitions…
Bhavesh Shah
  • 3,299
  • 11
  • 49
  • 73
1
vote
1 answer

Caused by: org.apache.kafka.common.record.InvalidRecordException: Incorrect declared batch size, records still remaining in file

We meet the problem when consume message from kafka-2.2.1 and We config compression.type = zstd on the kafka broker.The full exception stack is org.apache.kafka.common.KafkaException: Received exception when fetching the next record from test-10. If…
clearwater
  • 11
  • 3
1
vote
2 answers

How to find number of messages present in topic , number of partitions present for a topic using Java?

I have a kafka topic where I have to find number of messages present in a topic and number of partitions present in the topic using Java
MsSourDiesel
  • 31
  • 6
1
vote
0 answers

In what scenario, can kafka consumer lag spike all of sudden even when the production is zero?

We have a peculiar situation where we are tracking consumer lag and it is gradually going down and then it shoots all of a sudden, even when the production of messages in kafka topic has become zero. My understanding is that, let say even if no…
The-Proton-Resurgence
  • 808
  • 1
  • 13
  • 28
1
vote
1 answer

Kafka consumer logic for reading records from partitions

When a Kafka consumer reads from a its assigned partitions, is there any particular logic that is used by the consumer fetcher thread(s) to get the data out of the partitions? For example, does the fetcher thread make any logic/effort to read…
Mazen Ezzeddine
  • 662
  • 1
  • 8
  • 28
1
vote
1 answer

Kafka message key is null using Confluent.Kafka consumer library

I am using Debezium to perform CDC against my DB to create messages in Kafka. Using tools like kafdrop and OffsetExplorer I am able to see the message key and value. However, from a .NET framework application, using the Confluent.Kafka libraries,…
Kyley Jex
  • 53
  • 8
1
vote
1 answer

How does Kafka decide which records are contained in the consumer poll loop when there are more than `max.poll.records` records left?

I have a Kafka consumer group consuming several topics (each topic has more than one partition). All topics contain a considerable amount of records on each partition. I'm currently trying to make sense of the behavior when the consumer initially…
l7r7
  • 1,134
  • 1
  • 7
  • 23
1
vote
3 answers

kafka auto commit - reset consumer group in c#.net

I have topic and consumer group created for it. I started consumer with auto commit enabled. As my consumer is slow now there is lot of lag and as its auto commit the offset is stored. I have millions of messages and live data so no point is…
Mahesh Malpani
  • 1,782
  • 16
  • 27
1
vote
1 answer

How to assign more than 1 partition to a kafka consumer

I have created kafka consumer in c and created a topic with 10 partition, when i tried to read data using consumer it is only reading from 2 partitions and then says no more messages. I tried using both the methods i.e subscribe and assign but none…
Suraj
  • 11
  • 3
1
vote
1 answer

Kafka Consumer Invalid Payload Error Handler

I have the below configuration. When the message is invalid I want to send an email and for errors I want to save it in database. How can I handle this in errorHandler() ? @Configuration @EnableKafka public class KafkaConsumerConfig…
Pale Blue Dot
  • 511
  • 2
  • 13
  • 33
1
vote
0 answers

Nested Struct in a struct ksqldb

I am performing a create stream Select As statement which is joining a stream and table. The stream has a nested struct, and would like to take all the attributes defined in the stream to be joined with the table and created a nested structure as…
Kafka User
  • 13
  • 3
1
vote
0 answers

Kafka RETENTION.BYTES | Manual acknowledgement

In case CLEANUP policy is COMPACT,DELETE and our acknowledgement is MANUAL_IMMEDIATE, and we reach the retention.bytes -> while we were in middle of processing a message, once the logs are cleaned by cleanup policy, would the message be repicked? or…
Shreya Garg
  • 51
  • 7
1
vote
1 answer

Kafka error: Unknown partition (unknown_partition)

I am using https://github.com/zendesk/racecar (which uses https://github.com/appsignal/rdkafka-ruby under the hood, a Ruby wrapper for https://github.com/edenhill/librdkafka/) to consume a Kafka stream and every now and then I receive this error…
user1519240
  • 2,186
  • 1
  • 22
  • 20
1
vote
1 answer

Springboot kafka consumer not receiving messages after another has already received the message

I am new to KAFKA and would like help I have 2 applications (springboot), they are identical/copies only with different ports. http://localhost:8080/ http://localhost:8081/ They are both consumers the two listen to the topic XXX I have…
Peter Parker
  • 91
  • 1
  • 10
1
vote
1 answer

How to stop Preparing to rebalance group with old generation in Kafka?

I used Kafka for my web application and I found the below messages in kafka.log : [2021-07-06 08:49:03,658] INFO [GroupCoordinator 0]: Preparing to rebalance group qpcengine-group in state PreparingRebalance with old generation 105…
learner
  • 11
  • 1
  • 4
1 2 3
…
99
100