Kafka aborting producer batches due to fatal error

Question

I use Spring kafka for my application. It's working perfectly with spring boot version 2.3.2, but when I upgrade to 2.7.0, below error:

2022-05-27 15:32:33.357 "ubuntu1804" "core-service" [main,,] INFO  o.a.k.c.producer.KafkaProducer.configureTransactionState:532 - [Producer clientId=producer-1] Instantiated an idempotent producer.
2022-05-27 15:32:33.392 "ubuntu1804" "core-service" [main,,] INFO  o.a.k.c.s.a.AbstractLogin.login:61 - Successfully logged in.
2022-05-27 15:32:33.430 "ubuntu1804" "core-service" [main,,] INFO  o.a.k.c.utils.AppInfoParser.<init>:119 - Kafka version: 3.0.1
2022-05-27 15:32:33.432 "ubuntu1804" "core-service" [main,,] INFO  o.a.k.c.utils.AppInfoParser.<init>:120 - Kafka commitId: 8e30984f43e64d8b
2022-05-27 15:32:33.432 "ubuntu1804" "core-service" [main,,] INFO  o.a.k.c.utils.AppInfoParser.<init>:121 - Kafka startTimeMs: 1653640353428
2022-05-27 15:32:34.350 "ubuntu1804" "core-service" [kafka-producer-network-thread | producer-1,,] INFO  o.a.kafka.clients.Metadata.update:287 - [Producer clientId=producer-1] Cluster ID: XeVfqn0-QOOFk9uCnDkisw
2022-05-27 15:32:34.354 "ubuntu1804" "core-service" [kafka-producer-network-thread | producer-1,,] INFO  o.a.k.c.p.i.TransactionManager.transitionToFatalError:500 - [Producer clientId=producer-1] Transiting to fatal error state due to org.apache.kafka.common.errors.ClusterAuthorizationException: Cluster authorization failed.
2022-05-27 15:32:34.386 "ubuntu1804" "core-service" [kafka-producer-network-thread | producer-1,,] ERROR o.a.k.c.p.internals.Sender.maybeAbortBatches:497 - [Producer clientId=producer-1] Aborting producer batches due to fatal error
org.apache.kafka.common.errors.ClusterAuthorizationException: Cluster authorization failed.
2022-05-27 15:32:34.387 "ubuntu1804" "core-service" [kafka-producer-network-thread | producer-1,,] ERROR o.a.k.c.p.internals.Sender.maybeAbortBatches:497 - [Producer clientId=producer-1] Aborting producer batches due to fatal error
org.apache.kafka.common.errors.ClusterAuthorizationException: Cluster authorization failed.
2022-05-27 15:32:34.388 "ubuntu1804" "core-service" [kafka-producer-network-thread | producer-1,,] ERROR o.a.k.c.p.internals.Sender.maybeAbortBatches:497 - [Producer clientId=producer-1] Aborting producer batches due to fatal error
org.apache.kafka.common.errors.ClusterAuthorizationException: Cluster authorization failed.

Here is my kafka configuration:

spring:
  kafka:
    producer:
      bootstrap-servers: 10.98.0.1:9092,10.98.0.2:9092,10.98.0.3:9092
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.apache.kafka.common.serialization.StringSerializer
      properties:
        security:
          protocol: SASL_PLAINTEXT
        sasl:
          mechanism: PLAIN
          jaas:
            config: org.apache.kafka.common.security.plain.PlainLoginModule required username=${KAFKA_USERNAME:myuser} password=${ENV_KAFKA_PASSWORD:mypassword};

Here is my logging config

logging:
  level:
    org:
      springframework:
        web: INFO
      apache:
        kafka: INFO

When I set logging level to OFF, there is no error anymore. But I just want to know why and how to fix it. Thank you in advance

It's strange. It's authentication related. Since you turned off the logs, you are not seeing this error. Does your application work? — Gibbs, May 28 '22 at 16:10

score 3 · Accepted Answer · answered Jun 08 '22 at 10:26

3

It may be due to the version of the kafka client, which is not compatible with the kafka broker servion. Change the version of kafka client accordingly.

Other sources enter link description here

answered Jun 08 '22 at 10:26

bac tran

46
2

Upvoted ! . Thanks , this helped me to fix something I have been finding for a long time – voucher_wolves Aug 16 '22 at 14:34

Kafka aborting producer batches due to fatal error

1 Answers1