How to stop receiving kafka-metadata messages on a topic

Question

For some reason, sometimes, our Confluent Kafka consumer receives wierd messages on a specific topic:

{
  "magicByte": 2,
  "attributes": 0,
  "timestamp": "1623227829187",
  "offset": "11575814",
  "key": null,
  "value": {
    "type": "Buffer",
    "data": []
  },
  "headers": {},
  "isControlRecord": false,
  "batchContext": {
    "firstOffset": "11575814",
    "firstTimestamp": "1623227829187",
    "partitionLeaderEpoch": 5,
    "inTransaction": false,
    "isControlBatch": false,
    "lastOffsetDelta": 0,
    "producerId": "-1",
    "producerEpoch": -1,
    "firstSequence": -1,
    "maxTimestamp": "1623227829187",
    "timestampType": 0,
    "magicByte": 2
  }
}

I have consumed this exact same message multiple times even though I swallow all the errors and commit the offset even after failure.

It happends only on one specific topic. We deleted and re-created this topic from scarth and it didn't fix this issue.

In the following screenshot, we see that a message is empty while the other messages are not empty. What does it mean?

Producer - C#:

_producerBuilder = new ProducerBuilder<Null, string>(new ProducerConfig
                {
                    BootstrapServers = _kafka.BootstrapServers,
                    ClientId = Dns.GetHostName(),
                    LingerMs = _kafka.LingerMsKafka,
                    SaslMechanism = SaslMechanism.Plain,
                    SaslUsername = _kafka.SaslUsername,
                    SaslPassword = _kafka.SaslPassword,
                    SecurityProtocol = SecurityProtocol.SaslSsl,
                    EnableSslCertificateVerification = false
                }).Build();

Consumer NodeJs:

this.kafka = new Kafka.Kafka({
  brokers: [`${connectionOptions.host}:${connectionOptions.port}`],
  logLevel: Kafka.logLevel.NOTHING,
  ssl: !!connectionOptions.sasl,
  sasl: connectionOptions.sasl,
})
// ...
this.consumer = this.kafka.consumer({
  groupId: this.groupId,
})
// ...
await this.consumer.connect()
await this.consumer.subscribe({ topic: this.topic, fromBeginning: true })
await this.consumer.run({ batch }: EachBatchPayload) => {/* ...*/})

We are failing to consume the problematic message with ccloud: error - "panic: runtime error: index out of range [0] with length 0"

We are pretty lost. Any help will be great!

NodeJS: v14.15.4 KafkaJS: 1.16.0-beta.18

Do you have any idea what is producing those messages? Is it a producer using transactions because transactions write markers into the topic that have no body — OneCricketeer, Jun 09 '21 at 11:23
I have no idea what produce these msgs. even kafka-cli of confluent is crushing when it tries to read these msgs. — Stav Alfi, Jun 09 '21 at 11:25
I trust your ability to search elsewhere for that information, but it allows for idempotent producers and exactly once record delivery — OneCricketeer, Jun 09 '21 at 11:27
Ultimately, you cannot "stop it" because it's the producer that's writing the data. I'd suggest reaching out to Confluent Support to understand why the UI is able to show empty messages but the CLI cannot. Similarly, file an issue with KafkaJS project to see what can be done about it... Overall, the record has a unique timestamp and offset, just an empty byte array body, so there's nothing out of the ordinary about it — OneCricketeer, Jun 09 '21 at 11:33
Are you implying that the producer micro-service is sending me this data? or the C# client of the producer is sending me this data? — Stav Alfi, Jun 09 '21 at 13:21
I don't know what languages your services are in, but yes, there is _some client_ producing the data. Micro or not. And the `confluent-dotnet-kafka` client [can do transactions](https://github.com/confluentinc/confluent-kafka-dotnet#exactly-once-processing) — OneCricketeer, Jun 09 '21 at 22:18

score 0 · Accepted Answer · answered Jul 04 '21 at 21:43

0

This issue happen on kafka confluent (dev).

The support team did not solve this issue.

By moving to https://aiven.io/, this issue was never reproduced again.

answered Jul 04 '21 at 21:43

Stav Alfi

13,139
23
99
171

How to stop receiving kafka-metadata messages on a topic

1 Answers1