Kafka broker runs beautifully when running locally via docker-compose.
version: "3"
services:
zookeeper:
image: 'bitnami/zookeeper:latest'
ports:
- '2181:2181'
environment:
- ALLOW_ANONYMOUS_LOGIN=yes
kafka:
image: 'bitnami/kafka:latest'
ports:
- '9092:9092'
environment:
- KAFKA_BROKER_ID=1
- KAFKA_CFG_LISTENERS=PLAINTEXT://:9092
- KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092
- KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
- ALLOW_PLAINTEXT_LISTENER=yes
depends_on:
- zookeeper
When running via kubernetes though
apiVersion: v1
kind: Service
metadata:
name: zevrant-kafka-zookeeper-service
spec:
ports:
- port: 443
targetPort: 2181
selector:
app: zevrant-kafka-zookeeper-service
---
apiVersion: v1
kind: Service
metadata:
name: zevrant-kafka-service
spec:
type: NodePort
ports:Connection to node -1 (/<IP_ADDRESS>:30129) could not be established. Broker may not be available.
- port: 443
targetPort: 9092
nodePort: 30129
selector:
app: zevrant-kafka-service
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: zevrant-kafka-zookeeper-service-deployment
labels:
app: zevrant-kafka-zookeeper-service
spec:
replicas: 1
selector:
matchLabels:
app: zevrant-kafka-zookeeper-service
template:
metadata:
labels:
app: zevrant-kafka-zookeeper-service
spec:
volumes:
- name: zookeeper-data
nfs:
server: <IP_ADDRESS>
path: /zookeeper
imagePullSecrets:
- name: regcred
nodeSelector:
architecture: amd64
containers:
- name: zevrant-kafka-zookeeper-service
image: bitnami/zookeeper:3.7.0
env:
- name: ALLOW_ANONYMOUS_LOGIN
value: "yes"
ports:
- containerPort: 2181
volumeMounts:
- name: zookeeper-data
mountPath: /bitnami/zookeeper
- name: zevrant-kafka-zookeper-ui-service
image: elkozmon/zoonavigator:1.1.0
ports:
- containerPort: 9000
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: zevrant-kafka-service-daemonset
labels:
app: zevrant-kafka-service
spec:
replicas: 1
selector:
matchLabels:
app: zevrant-kafka-service
template:
metadata:
labels:
app: zevrant-kafka-service
spec:
volumes:
- name: kafka-storage
hostPath:
path: /opt/kafka/dev/
imagePullSecrets:
- name: regcred
nodeSelector:
architecture: amd64
containers:
- name: zevrant-kafka-broker-service
image: bitnami/kafka:2.8.0
volumeMounts:
- name: kafka-storage
mountPath: /bitnami/kafka
ports:
- containerPort: 9092
env:
- name: ALLOW_PLAINTEXT_LISTENER
value: "yes"
- name: KAFKA_CFG_ZOOKEEPER_CONNECT
value: "zevrant-kafka-zookeeper-service:443"
- name: KAFKA_BROKER_ID
value: "1"
- name: KAFKA_CFG_LISTENERS
value: "PLAINTEXT://0.0.0.0:9092"
- name: KAFKA_CFG_ADVERTISED_LISTENERS
value: "PLAINTEXT://<IP_ADDRESS>:30129"
The broker starts but repeatedly spams the error
INFO [Admin Manager on Broker 1]: Error processing create topic request CreatableTopic(name='zevrant-video-stream', numPartitions=1, replicationFactor=1, assignments=[], configs=[]) (kafka.server.ZkAdminManager)
I used the same command on both clusters to create the topic aside from the connection details and in both cases I ran the command from inside the running broker container.
Locally
kafka-topics.sh --create --topic zevrant-video-stream --zookeeper localhost:2181 --partitions 1 --replication-factor 1 --config retention.ms=86400000
Kubernetes
kafka-topics.sh --create --topic zevrant-video-stream --zookeeper zevrant-kafka-zookeeper-service:443 --partitions 1 --replication-factor 1 --config retention.ms=86400000
I'm not really sure why this works in one place and not the other.
Attempting to connect a producer to the kubernetes broker i receive
Connection to node -1 (/<IP_ADDRESS>:30129) could not be established. Broker may not be available.
When querying zookeeper for broker information, zookeeper shows the broker as connected
zkCli.sh get /brokers/ids/1
{"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://<IP_ADDRESS>:9092"],"jmx_port":-1,"features":{},"host":"<IP_ADDRESS>","timestamp":"1622398259745","port":9092,"version":5}
What would cause the broker to connected, online, but unavailable?
