Kafka brokers application property not recognised from docker-compose environment variables in a Spring Boot application

Question

I have a Spring Boot application that is configured to integrate with a Kafka instance. The Kafka instance runs as a container, through a docker-compose file. When I run the Spring Boot application, it picks up the Kafka brokers from the application properties file and the application runs fine.

The problem comes up when I package the application and create a docker image. After that I run the application using docker-compose. The Spring Boot application picks up all the security configurations, DB configurations from the environment variables in the docker-compose, but not for the Kafka brokers.

Here is the docker-compose that I am using to run Kafka:

version: '2'
services:
  zookeeper:
    container_name: Zookeeper
    image: confluentinc/cp-zookeeper:6.1.9
    ports:
      - "2181:2181"
    environment:
      - ZOOKEEPER_CLIENT_PORT=2181
      - ZOOKEEPER_SERVER_ID=1
  
  kafka:
    container_name: Kafka
    image: confluentinc/cp-kafka:6.1.9
    hostname: kafka
    ports:
      - "9092:9092"
    environment:
      - KAFKA_BROKER_ID=2
      - KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
      - KAFKA_ADVERTISED_LISTENERS=LISTENER_DOCKER_INTERNAL://kafka:19092,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9092
      - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
      - KAFKA_INTER_BROKER_LISTENER_NAME=LISTENER_DOCKER_INTERNAL
    depends_on:
      - zookeeper
    
  schemaregistry:
    container_name: Schema-Registry
    image: confluentinc/cp-schema-registry:6.1.9
    hostname: schemaregistry
    ports:
      - "8081:8081"
    environment:
      - SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS=PLAINTEXT://kafka:19092
      - SCHEMA_REGISTRY_HOST_NAME=schemaregistry
    depends_on:
      - kafka

Here is the Kafka properties/configuration in the application.yml file for the project:

spring:
  cloud:
    stream:
      kafka:
        binder:
          consumer-properties:
            key.deserializer: org.apache.kafka.common.serialization.StringDeserializer
            value.deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
            spring.json.trusted.packages: '*'
          producer-properties:
            key.serializer: org.apache.kafka.common.serialization.StringSerializer
            value.serializer: org.springframework.kafka.support.serializer.JsonSerializer
      bindings:
        deviceEvent-out-0:
          binder: kafka
          content-type: application/json

Here is the application-prod.yml file for the properties to be used for prod profile:

spring:
  cloud:
    stream:
      kafka:
        binder:
          auto-create-topics: true
          brokers: ${KAFKA_BROKERS}
          consumer-properties:
            max.poll.records: ${CONSUMER_MAX_POLL_RECORDS}
            schema.registry.url: ${SCHEMA_REGISTRY_URL}
          producer-properties:
            max.poll.records: ${PRODUCER_MAX_POLL_RECORDS}
            schema.registry.url: ${SCHEMA_REGISTRY_URL}
      bindings:
        produceMessage-out-0:
          destination: ${DEVICE_EVENT_TOPIC}

And finally, this the docker-compose file for running the application after its image is created:

version: '2'
services:
  devicemanagerms:
    container_name: MyApp
    image: myappms:latest
    ports:
      - "8097:8097"
    environment:
      - SPRING_PROFILES_ACTIVE=prod
      - SERVICE_LOG_LEVEL=DEBUG
      - SPRING_SECURITY_OIDC_ISSUER_URI=<URL for Keycloak>
      - OAUTH2_CLIENT_ID=microservice-client
      - OAUTH2_CLIENT_SECRET=<---MASKED--->
      - AWS_IOT_CONNECT_ENDPOINT=<---MASKED--->
      - AWS_IOT_CONNECT_CLIENT_ID=<---MASKED--->
      - SERVER_PORT=8097
      - POSTGRESQL_DATABASE_URL=<---MASKED--->
      - POSTGRESQL_DATABASE_USERNAME=<---MASKED--->
      - POSTGRESQL_DATABASE_PASSWORD=<---MASKED--->
      - POSTGRESQL_DATABASE_SCHEMA=<---MASKED--->
      - SCHEMA_REGISTRY_URL=http://192.168.29.92:8081
      - KAFKA_BROKERS=192.168.29.92:9092
      - CONSUMER_MAX_POLL_RECORDS=10
      - PRODUCER_MAX_POLL_RECORDS=10
      - DEVICE_EVENT_TOPIC=device_event

I am using:

mvn clean package -DskipTests jib:dockerBuild

to package the application and create an image.

The logs I am getting in the container when I run this application and try to publish a message to Kafka is:

WARN 1 --- [| adminclient-1] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-1] Connection to node 2 (/127.0.0.1:9092) could not be established. Broker may not be available.

which appears that the Kafka endpoint in the docker-compose environment variable is not picked up by the application and it is trying to use 127.0.0.1 which points to the container itself.

This application runs fine when I run it as a standalone app using mvn spring-boot:run.