Rsyslog hangs after memory usage 456 MB

Question

I am using rsyslog server to keep log of various server. Recently I have added 20 servers in rsyslog, after that rsyslog is frequently hangs the service (not receiving log until service restart). I observed the memory usage and when memory reaches 456MB then it hangs.

How can I get rid out of this problem. Rsyslog server has 16 GB RAM, it uses no more than 2 GB.

Hello Thiago,

Thanks for your quick reply,please find your required log.

sudo cat /var/log/messages | grep rsyslog

[root@rsyslog ~]# cat /var/log/messages | grep rsyslog
Sep  4 23:38:54 rsyslog rsyslogd: -- MARK --
Sep  5 11:25:08 rsyslog rsyslogd: -- MARK --
Sep  5 15:50:12 rsyslog kernel: imklog 5.8.10, log source = /proc/kmsg started.
Sep  5 15:50:12 rsyslog rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="10706" x-info="http://www.rsyslog.com"] start
Sep  5 15:50:12 rsyslog rsyslogd-3000: unknown priority name "" [try http://www.rsyslog.com/e/3000 ]
Sep  5 15:50:12 rsyslog rsyslogd: the last error occured in /etc/rsyslog.conf, line 3:"RSYSLOG_DEBUG="Debug NoStdOut""
Sep  5 15:50:12 rsyslog rsyslogd: warning: selector line without actions will be discarded
Sep  5 15:50:12 rsyslog rsyslogd-3000: unknown priority name "" [try http://www.rsyslog.com/e/3000 ]
Sep  5 15:50:12 rsyslog rsyslogd: the last error occured in /etc/rsyslog.conf, line 4:"RSYSLOG_DEBUGLOG="/var/log/syslog-debug""
Sep  5 15:50:12 rsyslog rsyslogd: warning: selector line without actions will be discarded
Sep  5 15:50:12 rsyslog rsyslogd-2124: CONFIG ERROR: could not interpret master config file '/etc/rsyslog.conf'. [try http://www.rsyslog.com/e/2124 ]
Sep  5 16:11:24 rsyslog kernel: imklog 5.8.10, log source = /proc/kmsg started.
Sep  5 16:11:24 rsyslog rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="10919" x-info="http://www.rsyslog.com"] start
Sep  5 16:11:24 rsyslog rsyslogd-3000: unknown priority name "" [try http://www.rsyslog.com/e/3000 ]
Sep  5 16:11:24 rsyslog rsyslogd: the last error occured in /etc/rsyslog.conf, line 3:"RSYSLOG_DEBUG="Debug NoStdOut""
Sep  5 16:11:24 rsyslog rsyslogd: warning: selector line without actions will be discarded
Sep  5 16:11:24 rsyslog rsyslogd-3000: unknown priority name "" [try http://www.rsyslog.com/e/3000 ]
Sep  5 16:11:24 rsyslog rsyslogd: the last error occured in /etc/rsyslog.conf, line 4:"RSYSLOG_DEBUGLOG="/var/log/syslog-debug""
Sep  5 16:11:24 rsyslog rsyslogd: warning: selector line without actions will be discarded
Sep  5 16:11:24 rsyslog rsyslogd-2124: CONFIG ERROR: could not interpret master config file '/etc/rsyslog.conf'. [try http://www.rsyslog.com/e/2124 ]
Sep  5 16:54:59 rsyslog kernel: imklog 5.8.10, log source = /proc/kmsg started.
Sep  5 16:54:59 rsyslog rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="11231" x-info="http://www.rsyslog.com"] start
Sep  5 16:54:59 rsyslog rsyslogd-3000: unknown priority name "" [try http://www.rsyslog.com/e/3000 ]
Sep  5 16:54:59 rsyslog rsyslogd: the last error occured in /etc/rsyslog.conf, line 3:"RSYSLOG_DEBUG="Debug NoStdOut""
Sep  5 16:54:59 rsyslog rsyslogd: warning: selector line without actions will be discarded
Sep  5 16:54:59 rsyslog rsyslogd-3000: unknown priority name "" [try http://www.rsyslog.com/e/3000 ]
Sep  5 16:54:59 rsyslog rsyslogd: the last error occured in /etc/rsyslog.conf, line 4:"RSYSLOG_DEBUGLOG="/var/log/syslog-debug""
Sep  5 16:54:59 rsyslog rsyslogd: warning: selector line without actions will be discarded
Sep  5 16:54:59 rsyslog rsyslogd-2124: CONFIG ERROR: could not interpret master config file '/etc/rsyslog.conf'. [try http://www.rsyslog.com/e/2124 ]
Sep  5 17:42:53 rsyslog kernel: imklog 5.8.10, log source = /proc/kmsg started.
Sep  5 17:42:53 rsyslog rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="11562" x-info="http://www.rsyslog.com"] start
Sep  5 17:42:53 rsyslog rsyslogd-3000: unknown priority name "" [try http://www.rsyslog.com/e/3000 ]
Sep  5 17:42:53 rsyslog rsyslogd: the last error occured in /etc/rsyslog.conf, line 3:"RSYSLOG_DEBUG="Debug NoStdOut""
Sep  5 17:42:53 rsyslog rsyslogd: warning: selector line without actions will be discarded
Sep  5 17:42:53 rsyslog rsyslogd-3000: unknown priority name "" [try http://www.rsyslog.com/e/3000 ]
Sep  5 17:42:53 rsyslog rsyslogd: the last error occured in /etc/rsyslog.conf, line 4:"RSYSLOG_DEBUGLOG="/var/log/syslog-debug""
Sep  5 17:42:53 rsyslog rsyslogd: warning: selector line without actions will be discarded
Sep  5 17:42:53 rsyslog rsyslogd-2124: CONFIG ERROR: could not interpret master config file '/etc/rsyslog.conf'. [try http://www.rsyslog.com/e/2124 ]
Sep  5 17:59:16 rsyslog kernel: fuse init (API version 7.14)
Sep  5 17:59:16 rsyslog seahorse-daemon[11835]: DNS-SD initialization failed: Daemon not running
Sep  5 17:59:16 rsyslog seahorse-daemon[11835]: init gpgme version 1.1.8
Sep  5 17:59:18 rsyslog polkitd[12048]: started daemon version 0.96 using authority implementation `local' version `0.96'
Sep  5 17:59:19 rsyslog vmusr[11909]: [ warning] [Gtk] Unable to locate theme engine in module_path: "clearlooks",
Sep  5 17:59:19 rsyslog vmusr[11909]: [ warning] [Gtk] Unable to locate theme engine in module_path: "clearlooks",
Sep  5 17:59:19 rsyslog vmusr[11909]: [ warning] [Gtk] Unable to locate theme engine in module_path: "clearlooks",
Sep  5 17:59:19 rsyslog vmusr[11909]: [ warning] [Gtk] Unable to locate theme engine in module_path: "clearlooks",
Sep  5 17:59:19 rsyslog vmusr[11909]: [ warning] [Gtk] Unable to locate theme engine in module_path: "clearlooks",
Sep  5 17:59:19 rsyslog vmusr[11909]: [ warning] [Gtk] Unable to locate theme engine in module_path: "clearlooks",
Sep  5 17:59:19 rsyslog vmusr[11909]: [ warning] [Gtk] Unable to locate theme engine in module_path: "clearlooks",
Sep  5 17:59:19 rsyslog vmusr[11909]: [ warning] [Gtk] Unable to locate theme engine in module_path: "clearlooks",
Sep  5 17:59:19 rsyslog vmusr[11909]: [ warning] [Gtk] Unable to locate theme engine in module_path: "clearlooks",
Sep  5 17:59:19 rsyslog vmusr[11909]: [ warning] [Gtk] Unable to locate theme engine in module_path: "clearlooks",
Sep  5 17:59:19 rsyslog vmusr[11909]: [ warning] [Gtk] Unable to locate theme engine in module_path: "clearlooks",
Sep  5 17:59:19 rsyslog vmusr[11909]: [ warning] [Gtk] Unable to locate theme engine in module_path: "clearlooks",
Sep  5 17:59:19 rsyslog vmusr[11909]: [ warning] [Gtk] Unable to locate theme engine in module_path: "clearlooks",
Sep  5 17:59:19 rsyslog vmusr[11909]: [ warning] [Gtk] Unable to locate theme engine in module_path: "clearlooks",
Sep  5 17:59:19 rsyslog vmusr[11909]: [ warning] [Gtk] Unable to locate theme engine in module_path: "clearlooks",
Sep  5 17:59:19 rsyslog vmusr[11909]: [ warning] [Gtk] Unable to locate theme engine in module_path: "clearlooks",
Sep  5 17:59:19 rsyslog vmusr[11909]: [ warning] [Gtk] Unable to locate theme engine in module_path: "clearlooks",
Sep  5 17:59:19 rsyslog vmusr[11909]: [ warning] [Gtk] gtk_disable_setlocale() must be called before gtk_init()
Sep  5 18:00:28 rsyslog ntpd[6805]: ntpd exiting on signal 15
Sep  5 18:00:28 rsyslog ntpd[12245]: ntpd 4.2.6p5@1.2349-o Sat Nov 23 18:21:48 UTC 2013 (1)
Sep  5 18:00:28 rsyslog ntpd[12246]: proto: precision = 0.061 usec
Sep  5 18:00:28 rsyslog ntpd[12246]: 0.0.0.0 c01d 0d kern kernel time sync enabled
Sep  5 18:00:28 rsyslog ntpd[12246]: Listen and drop on 0 v4wildcard 0.0.0.0 UDP 123
Sep  5 18:00:28 rsyslog ntpd[12246]: Listen and drop on 1 v6wildcard :: UDP 123
Sep  5 18:00:28 rsyslog ntpd[12246]: Listen normally on 2 lo 127.0.0.1 UDP 123
Sep  5 18:00:28 rsyslog ntpd[12246]: Listen normally on 4 eth0 fe80::250:56ff:feba:de61 UDP 123
Sep  5 18:00:28 rsyslog ntpd[12246]: Listen normally on 5 lo ::1 UDP 123
Sep  5 18:00:28 rsyslog ntpd[12246]: peers refreshed
Sep  5 18:00:28 rsyslog ntpd[12246]: Listening on routing socket on fd #22 for interface updates
Sep  5 18:00:28 rsyslog ntpd[12246]: 0.0.0.0 c016 06 restart
Sep  5 18:00:28 rsyslog ntpd[12246]: 0.0.0.0 c012 02 freq_set kernel -39.564 PPM
Sep  5 18:02:44 rsyslog init: tty (/dev/tty1) main process ended, respawning
Sep  5 18:02:53 rsyslog rsyslogd: -- MARK --
Sep  5 18:03:42 rsyslog ntpd[12246]: 0.0.0.0 c615 05 clock_sync
Sep  5 18:22:53 rsyslog rsyslogd: -- MARK --
Sep  5 18:42:53 rsyslog rsyslogd: -- MARK --
Sep  5 19:02:53 rsyslog rsyslogd: -- MARK --
Sep  5 19:22:53 rsyslog rsyslogd: -- MARK --
Sep  6 12:30:35 rsyslog kernel: imklog 5.8.10, log source = /proc/kmsg started.
Sep  6 12:30:35 rsyslog rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="18673" x-info="http://www.rsyslog.com"] start
Sep  6 12:50:35 rsyslog rsyslogd: -- MARK --

===================================================

rsyslog.conf file

rsyslog v5 configuration file

RSYSLOG_DEBUG="Debug NoStdOut"

RSYSLOG_DEBUGLOG="/var/log/syslog-debug"

For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html

If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

MODULES

$ModLoad imuxsock # provides support for local system logging (e.g. via logger command) $ModLoad imklog # provides kernel logging support (previously done by rklogd) $ModLoad immark # provides --MARK-- message capability $ModLoad ommysql

Provides UDP syslog reception

$ModLoad imudp $UDPServerRun 514

Provides TCP syslog reception

$ModLoad imtcp $InputTCPServerRun 514

GLOBAL DIRECTIVES

Use default timestamp format

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

File syncing capability is disabled by default. This feature is usually not required,

not useful and an extreme performance hit

$ActionFileEnableSync on

Include all config files in /etc/rsyslog.d/

$IncludeConfig /etc/rsyslog.d/*.conf

RULES

Log all kernel messages to the console.

Logging much else clutters up the screen.

kern.* /dev/console

kern.* /var/log/console

Log anything (except mail) of level info or higher.

Don't log private authentication messages!

*.info;mail.none;authpriv.none;cron.none /var/log/messages

The authpriv file has restricted access.

authpriv.* /var/log/secure

Log all the mail messages in one place.

mail.* -/var/log/maillog

Log cron stuff

cron.* /var/log/cron

Everybody gets emergency messages

*.emerg *

Save news errors of level crit and higher in a special file.

uucp,news.crit /var/log/spooler

Save boot messages also to boot.log

local7.* /var/log/boot.log

$AllowedSender TCP, 127.0.0.1, 192.**** $AllowedSender UDP, 127.0.0.1, 192.****

. :ommysql:127.0.0.1,rsyslogdb,rsyslog,mypassword

### begin forwarding rule

The statement between the begin ... end define a SINGLE forwarding

rule. They belong together, do NOT split them. If you create multiple

forwarding rules, duplicate the whole block!

Remote Logging (we use TCP for reliable delivery)

#

An on-disk queue is created for this action. If the remote host is

down, messages are spooled to disk and sent when it is up again.

$WorkDirectory /var/lib/rsyslog # where to place spool files

$ActionQueueFileName fwdRule1 # unique name prefix for spool files

$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)

$ActionQueueSaveOnShutdown on # save messages to disk on shutdown

$ActionQueueType LinkedList # run asynchronously

$ActionResumeRetryCount -1 # infinite retries if host is down

remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional

. @@remote-host:514

### end of the forwarding rule

A template to for higher precision timestamps + severity logging

$template SpiceTmpl,"%TIMESTAMP%.%TIMESTAMP:::date-subseconds% %syslogtag% %syslogseverity-text%:%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"

:programname, startswith, "spice-vdagent" /var/log/spice-vdagent.log;SpiceTmpl

Could you please edit your post and paste the output of `sudo cat /var/log/messages | grep rsyslog`? We should check for rsyslog error messages in this file. — Thiago Rider Augusto, Sep 05 '16 at 13:57
Do you have some log rotation configured? If yes, which configuration does it uses? — Thiago Rider Augusto, Sep 05 '16 at 14:02