1

I have a machine that aggregates syslog data from many devices, I keep all the devices that I want to have logging to my device in /etc/hosts so that they all have names that I understand. If a device logs to my machine that is not in /etc/hosts, I would like to have something else log so that I can find out what that host is and alert on it.

I have to take all the logs that are thrown at my devices, put them in one file and only minimally filter so sometimes I take on upwards of 12GB of logs per day. Because of the size of the log over time (which is rotated daily) I don't want to do something like cat/tr/cut/grep of the log in a cronjob.

Are there any other features in syslog-ng, linux tricks or iptables tweaks that would help identify when log messages come in from an unidentified host?

Peter Turner
  • 2,178
  • 9
  • 33
  • 45
  • Why can someone you don't know log to your machine to begin with? If you put the known machines into `/etc/hosts`, you could use this as a feed to whitelist those machines to the firewall... – Sven Aug 13 '18 at 21:25
  • @sven that's a fair point. The case here is that 9 times out of 10 it's misconfigured and it _should_ be in /etc/hosts. If I were a more strict sysadmin, I'd totally agree with you, but given the circumstances, I need to let the logs get logged, I just want to know when they happen. Then again, I could probably have an iptables rule for port 514 that is generated off /etc/hosts which logs one syslog per day for packets coming from unknown sources. – Peter Turner Aug 13 '18 at 21:54

1 Answers1

2

I'm not familiar with syslog-ng (I use rsyslog and others) but I think what you should really have is a map that takes in the source-ip or identifier, and returns what you want to call it, presumably there is a default you can use as well.

From https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-guide/60#TOPIC-956616

map-value-pairs: Rename value-pairs to normalize logs template and rewrite: Format, modify, and manipulate log messages > Modifying messages using rewrite rules > map-value-pairs: Rename value-pairs to normalize logs The map-value-pairs() parser allows you to map existing name-value pairs to a different set of name-value pairs. You can rename them in bulk, making it easy to use for log normalization tasks (for example, when you parse information from different log messages, and want to convert them into a uniform naming scheme). You can use the normal value-pairs expressions, similarly to value-pairs based destinations.

Available in syslog-ng OSE version 3.10 and later.

Declaration: parser parser_name {
    map-value-pairs(
        <list-of-value-pairs-options>
    ); };

And then used the output from that map in a template.

And from https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-guide/31#TOPIC-956502

destination d_file {
    file("/var/log/${YEAR}.${MONTH}.${DAY}/messages"
         template("${HOUR}:${MIN}:${SEC} ${TZ} ${HOST} [${LEVEL}] ${MESSAGE}\n")
         template-escape(no));
};
Cameron Kerr
  • 4,069
  • 19
  • 25