0

TLDR;

I'm getting significantly lower throughput when forwarding syslog messages with rsyslog on a Redhat server with 32 cores and 128Gi RAM to a remote server using TCP instead of UDP syslog in the omfwd action.

How can I use TCP for forwarding messages while keeping up with inbound message volume?

Note1: pps = packets per second

Note2: Note the difference in Rx vs Tx stats

When using UDP to forward incoming messages:

[~]$ ./check_network_stats.bash eth0
Rx    10088 pps Tx    10092 pps |--| Rx    7 Mbps Tx    7 Mbps
Rx    11858 pps Tx    11857 pps |--| Rx    8 Mbps Tx    8 Mbps
Rx    11503 pps Tx    11502 pps |--| Rx    8 Mbps Tx    8 Mbps
Rx    11423 pps Tx    11321 pps |--| Rx    8 Mbps Tx    8 Mbps

When using TCP to forward incoming messages:

[~]$ source check_network_stats.bash eth0
Rx    10318 pps Tx       87 pps |--| Rx    7 Mbps Tx    0 Mbps
Rx    12150 pps Tx      162 pps |--| Rx    8 Mbps Tx    0 Mbps
Rx     9504 pps Tx      139 pps |--| Rx    7 Mbps Tx    0 Mbps
Rx     9774 pps Tx       67 pps |--| Rx    6 Mbps Tx    0 Mbps
Rx    12894 pps Tx      159 pps |--| Rx    9 Mbps Tx    0 Mbps

rsyslog.conf:

# rsyslog configuration file


#################
#### MODULES ####
#################
         
module(load="imjournal" StateFile="imjournal.state")
module(load="imklog") # reads kernel messages (the same are read from journald)
module(load="immark" interval="300") # provides --MARK-- message capability
module(load="imudp" threads="4" batchSize ="128") # Provides UDP syslog reception
module(load="imptcp" threads="10") # Provides TCP syslog reception
module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat") # Use default timestamp format

###############
#### RULES ####
###############

ruleset(name="sendToLogstash") {
    action(type="omfwd"
       name="action_ls"
       Target="10.10.10.10"
       Port="514"
       Protocol="udp" # when this changed to TCP, throughput drops
       queue.type="FixedArray"
       )
}

###########################
#### LISTENERS         ####
###########################
input(type="imudp" port="514" ruleset="sendToLogstash")
input(type="imptcp" port="514" ruleset="sendToLogstash")

###########################
#### GLOBAL DIRECTIVES ####
###########################

mark.*  /var/log/messages

# Where to place auxiliary files
global(workDirectory="/var/lib/rsyslog")
Wood Techie
  • 1
  • I am not familiar with the `imptcp` module, but I've read a few posts about it having connection issues (I don't know if they've been resolved or not). You might want to try out the `imtcp` module. – eDonkey Jul 20 '22 at 09:10
  • Also, the queue type `FixedArray` is suggested only for a queue size of max. 10,000 messages. So you might aswell want to look into the [queues](https://www.rsyslog.com/doc/v8-stable/concepts/queues.html). – eDonkey Jul 20 '22 at 09:21

0 Answers0