1

We use syslog-ng to record metrics. We use systemd journal for logging, we added metrics as part of logs and then filtered by adding filter in /etc/syslog-ng.conf. This worked well but for certain process but if a process spams more log, due to default log suppression rate limit imposed by systemd, we used to drop metrics as well. We don't want to modify log suppression rate limit as that might impact CPU and performance. On other hand we didn't even want to loose metrics.

Wondering if there are some way to add some custom source in syslog-ng for this use case.

Naveen Singh
  • 81
  • 4

1 Answers1

1

Answering my own question, had to go through few places, then figured out that in syslog-ng we can add some custom source and use it for our usecase.

Added following to default /etc/syslog-ng.conf

 source metrics {
     unix-dgram("/run/metrics" flags(no-parse));
     # We can use stream socket as well
 };

destination metrics_priority_normal {
    file("/var/metrics/metrics_priority_normal" template("$MSG\n"));
 };

log {
    source(metrics);
    filter { match("MetricPriority=NORMAL") };
    destination(metrics_priority_normal);
};

Syslog-ng during start up will create now unix socket at /run/metrics, and we can directly log metrics there which will direct to /var/metrics/metrics_priority_normal

Example how to create client socket: https://man7.org/linux/man-pages/man7/unix.7.html

Server socket is taken care by syslog-ng.

One can add various other filters are well details: https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-guide#TOPIC-956384

Naveen Singh
  • 81
  • 4