0

I have a number of boxes that run docker containers. To keep a record of all the logs, we have 1 instance of logspout on every box that grabs all container logs on that box, and routes it to a syslog-ng instance (docker container) on a central host which stores it to a specific path.

I'm trying to extend this setup to use grafana/loki. Problem is that the __syslog_connection_hostname label always evaluates to "[local host fdqn].,[hostname],localhost.localdomain." no matter where the logs come from. I know syslog-ng knows where the actual hosts are, because the route that we store the logs are is something like: date_underscore/hostname/containername.log

I've tried setting keep-hostname(yes) in syslog-ng, no dice. Not sure what I'm missing.

syslog-ng conf

@version: 3.37
@include "scl.conf"
options {
  dir-perm(0755);
  keep-hostname(yes);
};
source s_network {
  default-network-drivers();
};
destination d_local {
  file("/logs/${YEAR}_${MONTH}_${DAY}/${HOST_FROM}/${PROGRAM}.log" perm(0755) create_dirs(yes));
};
destination d_loki {
  syslog("localhost" transport("tcp") port(1514));
};
log {
  source(s_network);
  destination(d_local);
  destination(d_loki);
};

promtail conf snippet

- job_name: syslog
  syslog:
    listen_address: 0.0.0.0:1514
    idle_timeout: 60s
    label_structured_data: yes
    labels:
      job: "syslog"
  relabel_configs:
    - source_labels: ['__syslog_message_hostname']
      target_label: 'container_name'
    - source_labels: ['__syslog_connection_hostname']
      target_label: 'hostname'
    - source_labels: ['__syslog_connection_ip_address']
      target_label: 'ip'
    - source_labels: ['__syslog_message_severity']
      target_label: 'severity'
WorkAppropriateUsername
  • 1

0 Answers0