I spent three days to find out where the problem is but without any success. The scenario is quite simple, I tried to deploy stack with grafana, grafana loki, promtail, kafka. I tried it with this docker-compose.yml:
version: "3.7"
networks:
kafka-net:
name: kafka-net
driver: bridge
services:
kafka_broker:
image: 'bitnami/kafka:latest'
container_name: kafka_broker
networks:
- kafka-net
ports:
- '9094:9094'
environment:
KAFKA_BROKER_ID: 1
KAFKA_ENABLE_KRAFT: 'yes'
KAFKA_CFG_PROCESS_ROLES: broker,controller
KAFKA_CFG_CONTROLLER_LISTENER_NAMES: CONTROLLER
KAFKA_CFG_LISTENERS: CLIENT://:29092,CONTROLLER://:9093,EXTERNAL://:9094
KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,CLIENT:PLAINTEXT,EXTERNAL:PLAINTEXT
KAFKA_CFG_INTER_BROKER_LISTENER_NAME: CLIENT
KAFKA_CFG_ADVERTISED_LISTENERS: CLIENT://kafka_broker:29092,EXTERNAL://localhost:9094
KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: 1@127.0.0.1:9093
KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'
KAFKA_DEFAULT_REPLICATION_FACTOR: 1
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
ALLOW_PLAINTEXT_LISTENER: 'yes'
kafka-ui:
image: provectuslabs/kafka-ui:latest
container_name: kafka-ui
networks:
- kafka-net
ports:
- '8080:8080'
environment:
KAFKA_CLUSTERS_0_NAME: "local"
KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: "kafka_broker:29092"
depends_on:
- "kafka_broker"
loki:
image: grafana/loki:latest
container_name: loki
ports:
- "3100:3100"
command: -config.file=/etc/loki/local-config.yaml
networks:
- kafka-net
promtail:
image: grafana/promtail:latest
container_name: promtail
volumes:
- /var/lib/docker/containers:/var/lib/docker/containers
- /var/workspace/promtail/docker-config.yml:/etc/promtail/docker-config.yml
command: -config.file=/etc/promtail/docker-config.yml
networks:
- kafka-net
grafana:
image: grafana/grafana:latest
container_name: grafana
ports:
- "3000:3000"
networks:
- kafka-net
Everything worked fine. Then I wanted to deploy it with ansible through Jenkins to our dev environments. The environment is based on Linux ubuntu. Nothing special because the same flow is tested many times deploying our services. The deployment does not include docker-compose, it is just about individual podman containers.
The loki config looks like this:
auth_enabled: false
server:
http_listen_port: 3100
grpc_listen_port: 9096
common:
path_prefix: /tmp/loki
storage:
filesystem:
chunks_directory: /tmp/loki/chunks
rules_directory: /tmp/loki/rules
replication_factor: 1
ring:
instance_addr: 127.0.0.1
kvstore:
store: inmemory
schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 24h
ruler:
alertmanager_url: http://localhost:9093
analytics:
reporting_enabled: false
The promtail looks like this:
server:
http_listen_address: 0.0.0.0
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
clients:
- url: http://localhost:3100/loki/api/v1/push
scrape_configs:
- job_name: varlogs
static_configs:
- targets: [localhost]
labels:
job: varlogs
__path__: /var/logtest/system.log
- job_name: kafka
kafka:
brokers:
- kafka_broker:29092
topics:
- grafana
labels:
job: kafka
host: ${HOSTNAME:promtail}
relabel_configs:
- action: replace
source_labels:
- __meta_kafka_topic
target_label: topic
- action: replace
source_labels:
- __meta_kafka_partition
target_label: partition
- action: replace
source_labels:
- __meta_kafka_group_id
target_label: group
I created podman bridge network called mycustom-bridge-net.
kafka-env.env file:
KAFKA_BROKER_ID=1
KAFKA_ENABLE_KRAFT=yes
KAFKA_CFG_PROCESS_ROLES=broker,controller
KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
KAFKA_CFG_LISTENERS=CLIENT://:29092,CONTROLLER://:9093,EXTERNAL://:9094
KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,CLIENT:PLAINTEXT,EXTERNAL:PLAINTEXT
KAFKA_CFG_INTER_BROKER_LISTENER_NAME=CLIENT
KAFKA_CFG_ADVERTISED_LISTENERS=CLIENT://kafka_broker:29092,EXTERNAL://localhost:9094
KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@127.0.0.1:9093
KAFKA_AUTO_CREATE_TOPICS_ENABLE=false
KAFKA_DEFAULT_REPLICATION_FACTOR=1
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
ALLOW_PLAINTEXT_LISTENER=yes
The podman commands:
podman run -d --env-file=/kafka/env/kafka-env.env -v /kafka/data:/bitnami/kafka --name kafka_broker --net=mycustom-bridge-net bitnami/kafka:latest
podman run -d -p 3100:3100 -v /grafana-loki/data:/loki -v /grafana-loki/config/local-config.yml:/etc/loki/local-config.yaml --name loki --net=mycustom-bridge-net grafana/loki:latest -config.file=/etc/loki/local-config.yaml
podman run -d -v /var/lib/containers:/var/lib/containers -v /grafana-promtail/config/docker-config.yml:/etc/promtail/docker-config.yml --name promtail --net=mycustom-bridge-net grafana/promtail:latest -config.file=/etc/promtail/docker-config.yml
podman run -d -p 3000:3000 -v /grafana/data:/var/lib/grafana --name grafana --net=mycustom-bridge-net grafana/grafana:latest
I have to say, if there is any mistake in these configuration files it is probably typo because I deployed every single service mentioned here. Communication between Kafka and Promtail is running well. I saw push into Loki from Promtail as well, BUT when I wanted to connect into Loki through grafana datasource I got this:
Unable to fetch labels from Loki (Failed to call resource), please check the server logs for more details
I tried to call Loki health check from browser http://IP_ADDRESS:3100/ready and I got ready. Every time when I tried to call from the browser the labels endpoint "/loki/api/v1/labels" it returns nothing mostly it timed out.
EDIT BUT when I tried to call Loki's health check by curl:
curl http://localhost:3100/ready
from Kafka container, I get 503 - Service unavailable. I guess there is a wrong network setup between these containers within network mycustom-bridge-net.
CONCLUSION: Does anyone know why I cannot get the labels from Loki ?
