rsyslog does not discard messages

Question

I have the following rsyslog.conf:

$PreserveFQDN on
# rsyslog v5 configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imklog   # provides kernel logging support (previously done by rklogd)
#$ModLoad immark  # provides --MARK-- message capability

# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514

# Provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514


#### GLOBAL DIRECTIVES ####

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf


#### RULES ####

#discard loadbalancer messages
:msg, contains, "default send string" ~

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog


# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 *

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log


# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/lib/rsyslog # where to place spool files
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList   # run asynchronously
#$ActionResumeRetryCount -1    # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
#

*.* @localhost:5000
# ### end of the forwarding rule ###

Which contains the following rule for discarding a specific message:

#discard loadbalancer messages
:msg, contains, "default send string" ~

It does not seem to do what it is supposed to do, I can still see these messages in /var/log/messages:

[~] # tail -f /var/log/messages
Dec  3 02:51:38 default send string
Dec  3 02:51:42 default send string
Dec  3 02:51:43 default send string
Dec  3 02:51:47 default send string
Dec  3 02:51:48 default send string
Dec  3 02:51:52 default send string
Dec  3 02:51:53 default send string
Dec  3 02:51:57 default send string
Dec  3 02:51:58 default send string
Dec  3 02:52:02 default send string
Dec  3 02:52:03 default send string

Please note that this is a Logstash server which receives syslog messages from other servers to port 514 and forwards them to its own port 5000 in order for them to be processed by Logstash, that's why there is this line in the config file:

 *.* @localhost:5000

Answer 1

You need to make sure that your rule for discarding the message should be called before the one that writes it in a file.

By default, Rsyslog have a /etc/rsyslog.d/50-default.conf that writes messages as it is. Maybe putting your rule before:

$IncludeConfig /etc/rsyslog.d/*.conf

should work.

And Rsyslog warns to not use ~, you should use stop instead.

Answer 2

I was thinking the hostname is "default" and the :msg is "send string", but I couldn't get it to filter that way either. So, I did some more searching and found the following:

:rawmsg, isequal, "default send string" stop

That finally made the messages go away. For what its worth, they appear to be from our load balancer, probably a syslog/udp healthcheck.

Reference: http://lists.adiscon.net/pipermail/rsyslog/2012-September/030562.html