I have docker swarm with 3 node with following stack file
version: '3.2'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
deploy:
mode: global
volumes:
- /zookeeper/data:/data
- /zookeeper/datalog:/datalog
environment:
ZOO_MY_ID: 1
ZOO_PORT: 2181
ZOO_SERVERS: server.1=zookeeper:2888:3888
kafka:
image: wurstmeister/kafka:latest
ports:
- target: 9094
published: 9094
protocol: tcp
mode: host
deploy:
mode: global
environment:
HOSTNAME_COMMAND: "docker info | grep 'Node Address:' | cut -d' ' -f 5"
BROKER_ID_COMMAND: "{{`docker info -f ''{{.Swarm.NodeAddr}}'' | cut -d . -f 4`}}"
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://_{HOSTNAME_COMMAND}:9094
KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9094
KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ./kafka:/var/lib/kafka/data
When i am trying to connect from filebeat producer with hostIP1:9094, hostIP2:9094, hostIP3:9094, getting error
2021-06-22T13:22:01.215+0530 DEBUG [kafka] kafka/client.go:385 Kafka publish failed with: dial tcp: lookup f0038421e470: no such host
It means following command returning container id to filebeat as a host IP, not actual host IP.
HOSTNAME_COMMAND: "docker info | grep 'Node Address:' | cut -d' ' -f 5"
Second issue with BROKER_ID_COMMAND, its giving following error at kafka , But if i run on linux machine it will give last digit of IP for broker ID
org.apache.kafka.common.config.ConfigException: Invalid value for configuration broker.id: Not a number of type INT
