AWS MSK gives warning

Question

we are working on spring boot application and we have implemented MSK AWS (Kafka) , producer and consumer communication is working as expected but once I checked logs it is giving warning as below.

Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available

Spring boot yml configuration file

spring:
   cloud:
     config:
       enabled: true
   application:
      name: my-service
   kafka:
     producer:
       bootstrap-servers: broker1MSKUrl:9092,broker2Url:9092
     consumer:
       bootstrap-servers: broker1MSKUrl:9092,broker2Url:9092

Spring boot kafka consumer class

@Value(value = "${spring.kafka.consumer.bootstrap-servers}")
private String bootstrapAddress;

    Map<String, Object> props = new HashMap<>();
    props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, 
    bootstrapAddress);
    props.put(ConsumerConfig.GROUP_ID_CONFIG, groupId);
    props.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, 
    bootstrapAddress);
    props.put(JsonDeserializer.TRUSTED_PACKAGES, "*");
    AdminClient adminClient = AdminClient.create(props);
    ListTopicsOptions listTopicsOptions = new ListTopicsOptions();
    listTopicsOptions.listInternal(true);

we are wondering why we are getting localhost warning while we have connected to AWS MSK

This is our MSK configuration

auto.create.topics.enable=true
default.replication.factor=3
min.insync.replicas=2
num.io.threads=8
num.network.threads=5
num.partitions=1
num.replica.fetchers=2
replica.lag.time.max.ms=30000
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
socket.send.buffer.bytes=102400
unclean.leader.election.enable=true
zookeeper.session.timeout.ms=18000

check here please -> https://stackoverflow.com/a/65213076/2165146 — muhammed ozbilici, Aug 23 '22 at 10:39

OneCricketeer · Accepted Answer · 2022-09-06T14:18:26.807

1

You're creating an AdminClient. So you need to set spring.kafka.admin.bootstrap-servers, not just the consumer/producer. Otherwise, the admin client will default to localhost.

Or use spring.kafka.bootstrap-servers to see all three client types.

Also, admin client doesn't use GROUP_ID_CONFIG or TRUSTED_PACKAGES... Try to limit what properties you set to specific clients. For example, bootstrap servers can come from CommonClientConfigs class, but that's also not necessary as Spring Boot will auto-wire the property mentioned above.

edited Sep 06 '22 at 14:18

answered Aug 23 '22 at 18:58

OneCricketeer

179,855
19
132
245

we are not using locahost:9092 I have posted my code, props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapAddress); in bootstrapAddress variable we are passing below values as brokers b-1.xxxx.xxxxx.kafka.us-xxx-2.amazonaws.com:9092,b-2.xxxxx.xxxx.c12.kafka.us-xxx-x.amazonaws.com:9092 as I mentioned in my question producer and consumer working correctly , we are posting our message in topic which is available in MSK's cluster but my main concern is it is printing lots of warning (localhost/127.0.0.1:9092) could not be established. Broker may not be available – Kapil Devmurari Aug 24 '22 at 04:28
The logs aren't wrong, and the code you've provided doesn't show the actual variable value. I don't know what else to tell you. More specifically, you've only set the AdminClient configuration bootstrap servers, not the producer or consumer. If you're using Spring Boot, you should set `spring.kafka.bootstrap.servers` in your properties file, which will set all the Kafka clients. (Can you edit the question to show it?). **The default value for that property is localhost:9092**. You also need to show more of the exception to see which specific client is throwing the error – OneCricketeer Aug 24 '22 at 13:14
I have updated my question and mentioned application yml file can you please suggest something .. thank you in advance – Kapil Devmurari Aug 25 '22 at 06:19

AWS MSK gives warning

1 Answers1