Nginx cannot connect fluentd in EFK stack

Question

I am setting up a stack with an application consisting of nginx, redis, mysql, myapp. Nginx proxies requests to myapp. I want to send logs from nginx to EFK stack, but an error occurs when starting the nginx service:

Error response from daemon: dial tcp 127.0.0.1:24224: connect: connection refused

docker-compose.yml for stack with myapp

version: "3.8"
services:
  
  nginx:
    image: nginx:alpine
    deploy:
      mode: replicated
      replicas: 2
      labels:
        - traefik.enable=true
        - traefik.http.routers.node1.rule=Host(`${NODE1}`)
        - traefik.http.routers.node1.service=nginx
        - traefik.http.routers.node2.rule=Host(`${NODE2}`)
        - traefik.http.routers.node2.service=nginx
        - traefik.http.routers.node3.rule=Host(`${NODE3}`)
        - traefik.http.routers.node3.service=nginx
        - traefik.http.services.nginx.loadbalancer.server.port=80
      placement:
        constraints:
          - node.role == manager
    logging:
      driver: fluentd
      options:
        fluentd-address: localhost:24224
        tag: nginx-
    volumes:
      - ./nginx/default.conf:/etc/nginx/conf.d/default.conf
    ports: 
      - 80:80
    depends_on: 
      - myapp
    networks:
      - traefik-public
...

All stacks are in the same traefik-public network, if you make ping fluentd from any container, fluentd responds

Part of efk.yml

version: "3.7"
services:

  fluentd:
    image: registry.rebrainme.com/docker_users_repos/3912/dkr-30-voting/fluentd
    deploy:
      mode: global
    volumes:
      - /mnt/fluent.conf:/fluentd/etc/fluent.conf
    ports:
      - "24224:24224"
      - "24224:24224/udp"
    depends_on:
      - elasticsearch
      - kibana
    networks:
      - traefik-public
...

fluent.conf

<source>
  @type forward
  port 24224
  bind localhost
</source>

<match *.**>
  @type copy

  <store>
    @type elasticsearch
    host elasticsearch
    port 9200
    logstash_format true
    logstash_prefix fluentd
    logstash_dateformat %Y%m%d
    include_tag_key true
    type_name access_log
    tag_key @log_name
    flush_interval 1s
  </store>

  <store>
    @type stdout
  </store>
</match>

I ask for help

In the `docker-compose.yml` file, you are providing the address to the fluentd service. But You give `fluentd-address: localhost:24224` shouldn't it be `fluentd-address: fluentd:24224` — Paulo, Jun 08 '22 at 21:18
I tried to specify the name of the service, then an error occurred when trying to connect: `Error response from daemon: dial tcp: lookup fluentd: Temporary failure in name resolution` — Rostislav Udaltsov, Jun 09 '22 at 03:16
Hmmm, okay. Could it be because you use comoose files ? Why don't you merge docker-compose.yml and efk.yml ? — Paulo, Jun 09 '22 at 05:36
Combined everything into one yml file, everything worked! But how to achieve the same effect with two stacks? — Rostislav Udaltsov, Jun 09 '22 at 08:24
Then I think this is another question. You should look for somehting like share context between multiple compose file. or somehting — Paulo, Jun 09 '22 at 12:14

score 0 · Answer 1 · answered Jun 09 '22 at 12:15

Tldr;

Because you are using 2 compose files. docker-compose.yml and efk.yml They are not sharing the same value for the network.

To Fix

Combine both file in a single one.

To Fix (with still 2 files separated)

You should first off all create a network.

docker network create traefik-public

Then update both compose file with

networks: 
  default: 
    external: 
      name: traefik-public

This should make it work.

score 0 · Accepted Answer · answered Jun 14 '22 at 06:19

In order for there to be a connection between the efk stack and the application stack, it is necessary to bind the fluentd port to the host port

version: "3.7"
services:

  fluentd:
    image: my_fluentd_image:latest
    deploy:
      mode: global
    configs:
      - source: fluent-conf
        target: /fluentd/etc/fluent.conf
    ports:
      - target: 24224
        published: 24224
        protocol: tcp
        mode: host
    depends_on:
      - elasticsearch
      - kibana
    networks:
      - traefik-public

Nginx cannot connect fluentd in EFK stack

2 Answers2

Tldr;

To Fix

To Fix (with still 2 files separated)