0

I have a problem with a docker-compose in swarm. This is my docker-compose:

    version: "3"



services:

  zookeeper:
    image: 'bitnami/zookeeper:latest'
    deploy:
      replicas: 1
    ports:
      - "2181:2181"
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
    networks:
      - ertest

  kafka:
    image: 'bitnami/kafka:latest'
    deploy:
      replicas: 1
    ports:
      - 9093:9092
    depends_on:
      - zookeeper
    environment:
      KAFKA_CFG_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_CFG_LISTENERS: INTERNAL://:9093,OUTSIDE://:9092
      KAFKA_CFG_ADVERTISED_LISTENERS: INTERNAL://kafka:9093,OUTSIDE://sub.domain.ltd:9092
      KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_CFG_INTER_BROKER_LISTENER_NAME: INTERNAL
      ALLOW_PLAINTEXT_LISTENER: "yes"
    networks:
      - ertest

  igi-interface:
    image: nsdevuti01.netstudio.it:8083/er/igi-interface:test
    depends_on:
      - kafka
    ports:
      - 8091:8091
    environment:
      - MP_MESSAGING_CONNENCTOR_SMALLRYE-KAFKA_SSL_TRUSTSTORE_LOCATION=/deployments/ssh/igi-gateway.server.truststore.jks
      - MP_MESSAGING_CONNENCTOR_SMALLRYE-KAFKA_SSL_TRUSTSTORE_PASSWORD=changeit
      - MP_MESSAGING_CONNENCTOR_SMALLRYE-KAFKA_SSL_KEYSTORE_LOCATION=/deployments/ssh/igi-gateway.server.keystore.jks
      - MP_MESSAGING_CONNENCTOR_SMALLRYE-KAFKA_SSL_KEYSTORE_PASSWORD=changeit
      - KAFKA_BOOTSTRAP_SERVERS=kafka:9092
    volumes:
      - ./config/igi-gateway:/deployments/ssh
    networks:
      - ertest

networks:
  ertest:
    driver: overlay
  ingress:
    external: true

The services are up (for this try they are deployed on the same node), but i receive the following error when logging the igi-interface related container:

14:54:46 WARN  [or.ap.ka.cl.NetworkClient] (kafka-producer-network-thread | kafka-producer-user-message) [Producer clientId=kafka-producer-user-message] Bootstrap broker kafka:9092 (id: -1 rack: null) disconnected
14:54:46 WARN  [or.ap.ka.cl.NetworkClient] (kafka-producer-network-thread | kafka-producer-account-message) [Producer clientId=kafka-producer-account-message] Connection to node -1 (kafka/10.0.1.10:9092) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue.

Is that a problem with the kafka container generated with the image bitnami/kafka?

this is the log of the kafka service:

[2022-05-27 15:05:53,780] WARN [SocketServer listenerType=ZK_BROKER, nodeId=1001] Unexpected error from /10.0.1.66 (channelId=10.0.1.70:9092-10.0.1.66:37934-372); closing connection (org.apache.kafka.common.network.Selector)

org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 369296129 larger than 104857600)

at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:105)

at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452)

at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402)

at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674)

at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576)

at org.apache.kafka.common.network.Selector.poll(Selector.java:481)

at kafka.network.Processor.poll(SocketServer.scala:1144)

at kafka.network.Processor.run(SocketServer.scala:1047)

at java.base/java.lang.Thread.run(Thread.java:829)

these are the container on the network created:

"Containers": {
        "05b840fca9768baa737f251b8687027c7c7129e629d9edfab1c79eaab019bc79": {
            "Name": "test_igi-interface.1.gf1zbtaajohfhj09370mugqsn",
            "EndpointID": "299107c675dc38275f8a4eda6c6e94155c9047bbada7740b12f60170b9fe96ea",
            "MacAddress": "02:42:0a:00:01:48",
            "IPv4Address": "10.0.1.72/24",
            "IPv6Address": ""
        },
        "6285eb948cfedf917da28982ed08c8105712a3d6f6b0159b18e0606110320508": {
            "Name": "test_kafka.1.jpawj3nwf2q68niev5be7h8dm",
            "EndpointID": "7700a38302942549f464b99bfc48ed49e6d216d1d5e744368c65c0385cdfe2b9",
            "MacAddress": "02:42:0a:00:01:46",
            "IPv4Address": "10.0.1.70/24",
            "IPv6Address": ""
        },
        "a7f8fde2f5bcdd2abb6f74447b6922cc9d1f694bbc9f9c085935277bbf7404c3": {
            "Name": "test_zookeeper.1.bumghhajiejes9z1c9x3r4aak",
            "EndpointID": "5b2cf22296a4e85b6d83a37ca2b5ff93a3f29a5daf9e20cd45495097b3bcb773",
            "MacAddress": "02:42:0a:00:01:4a",
            "IPv4Address": "10.0.1.74/24",
            "IPv6Address": ""
        },
        "lb-test_ertest": {
            "Name": "test_ertest-endpoint",
            "EndpointID": "ed8f4e31d95fc4d9090010cd0d185e708183f7ceaf0738e3055fe7fc378f6fd5",
            "MacAddress": "02:42:0a:00:01:42",
            "IPv4Address": "10.0.1.66/24",
            "IPv6Address": ""
        }
    }

Kafka have an issue with test_ertest-endpoint, but i don't know why. How can i fix this problem? the container are under the same network (ertest) and this is an overlay network.

Thanks for your time.

Fabio_F
  • 1
  • Where are you actually using `sub.domain.ltd:9092`? The error says you tried to connect to `kafka:9092`, and then your client actually got `10.0.1.66` which isn't a Kafka server, so you get `InvalidReceiveException` because the server and port _responded_, but it was not the Kafka protocol that is responding. https://www.confluent.io/blog/kafka-listeners-explained/ – OneCricketeer May 27 '22 at 17:04
  • Also the port mapping from 9093 to 9092 will cause issues as well. More specifically, the external host wouldn't accept traffic on port 9092, which is the port being returned as the "OUTSIDE" listener – OneCricketeer May 27 '22 at 22:27
  • Hi @OneCricketeer, thanks for the reply. I have corrected the error (copypaste error) but i have the same issues. I have changed the wrong environment variables with these: KAFKA_CFG_LISTENERS=CLIENT://:9092,EXTERNAL://0.0.0.0:9093 KAFKA_CFG_ADVERTISED_LISTENERS=CLIENT://kafka:9092,EXTERNAL://kafka:9093 i've included this for the remote connection, but it didn't work. i"nclude EXTERNAL://0.0.0.0:9093 in KAFKA_CFG_LISTENERS to allow for remote connections." this is the link for the guide: https://github.com/bitnami/bitnami-docker-kafka/blob/master/README.md Do i miss something? – Fabio_F May 30 '22 at 15:02
  • Note: i've exposed the port 9093:9093 and 9092:9092 on the Kafka service in docker-compose for the swarm. – Fabio_F May 30 '22 at 15:20
  • This are the environment variables i had set on the Kafka service: environment: - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181 - ALLOW_PLAINTEXT_LISTENER=yes - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CLIENT:PLAINTEXT,EXTERNAL:PLAINTEXT - KAFKA_CFG_LISTENERS=CLIENT://:9092,EXTERNAL://0.0.0.0:9093 - KAFKA_CFG_ADVERTISED_LISTENERS=CLIENT://kafka:9092,EXTERNAL://kafka:9093 - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=CLIENT – Fabio_F May 30 '22 at 15:38
  • Please [edit] your question with the values rather than a comment. With those settings, you have the exact same advertised listener twice on two different ports, which is pointless. In any case, I've never used Kafka with Docker Swarm, but you're you say you are on one host, so that shouldn't matter, and the blog linked in the first comment details all the network debugging you should need. Use netcat and ping to test IP's, Docker service names, and ports. Use kcat to check Kafka listener values between services. – OneCricketeer May 31 '22 at 14:48

0 Answers0