1

In local Kafka messages are consuming successfully with the below application configuration

spring:
  cloud:
    stream:
      kafka:
        binder:
          replicationFactor: 1
          auto-create-topics: true
          brokers: localhost:9092
      bindings:
        binding-in-sse:
          destination: sse-topic
          content-type: text/plain
          group: earlywage
        binding-out-sse:
          destination: sse-topic
          content-type: text/plain
          group: earlywage

Now in Dev need to configure with msk. The MSK configurations are as follows :

3 Partitions, 3 Replicas, 2 Brokers, SASL/SCRAM authentication, retention.ms=604800000, max.message.bytes=2097164.

vpc is private and trying to connect inside same vpc with the following config.

spring:
  cloud:
    stream:
      kafka:
        binder:
          replicationFactor: 1
          auto-create-topics: true
          brokers:
            - b-1.****.***.c2.kafka.REGION.amazonaws.com:PORT
            - b-2.****.***.c2.kafka.REGION.amazonaws.com:PORT
          configuration:
            security.protocol: SASL_PLAINTEXT
            sasl.mechanism: SCRAM-SHA-512
            sasl:
              jaas:
                config: org.apache.kafka.common.security.scram.ScramLoginModule required username="***" password="*****";
      bindings:
        binding-in-sse:
          destination: sse-topic
          content-type: text/plain
          group: earlywage
        binding-out-sse:
          destination: sse-topic
          content-type: text/plain
          group: earlywage

What is the correct config required to consume the message from the brokers. Thank you.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Manoj Kumar
  • 21
  • 2
  • You should use IAM with MSK, but config looks okay to me, what specific error are you getting? – OneCricketeer Jun 28 '23 at 12:53
  • It is failing during authentication. The following is the error which I'm getting. – Manoj Kumar Jun 29 '23 at 13:17
  • [AdminClient clientId=adminclient-1] Connection to node -1 terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0. (2) Firewall blocking Kafka TLS traffic (e.g., it may only allow HTTPS traffic). (3) Transient network issue. – Manoj Kumar Jun 29 '23 at 13:18
  • I suggest contacting MSK support. We can't help you here without being part of your network or knowing what VPC rules you've configured. https://docs.aws.amazon.com/msk/latest/developerguide/client-access.html – OneCricketeer Jun 29 '23 at 14:36

0 Answers0