5

I have a service set up on CentOS 7 which outputs a lot of log messages, which results in this in the system log / journal: systemd-journal[xxxx]: Suppressed 864 messages from /system.slice/example.service.

However, I want to see all of the service messages for debug and diagnostic purposes. Furthermore, I want to disable rate limiting, since this is just a test server.

I've set RateLimitInterval=0 and RateLimitBurst=0 in /etc/systemd/journald.conf, but systemd-journal doesn't seem to obey this setting. A systemctl restart systemd-journald command or system restart doesn't seem to work.

Any suggestions on how to diagnose this? Is there a command to show what configuration systemd-journald is currently using?

David
  • 51
  • 1
  • 2
  • did you reload systemd configuration itself? systemctl daemon-reload – Dennis Nolte Jan 04 '19 at 08:34
  • @DennisNolte, no luck, I even did a `daemon-reload`, `restart systemd-journald`, followed by another `daemon-reload` just to make sure command order wasn't an issue. Thank you for the suggestion. Please let me know if you come up with anything else. – David Jan 06 '19 at 23:12
  • 1
    I decided to temporarily work around the problem by redirecting the output to a file via the service's ExecStart setting. – David Jan 07 '19 at 01:28
  • Systemd maintainer stated it's considered to be a bug in *the kernel* if many messages are created in a short interval. – U. Windl Jun 01 '21 at 13:00
  • 1
    Would be nice to have logger that focuses on logging instead of going on strike due to "bad working conditions" – Jonas Berlin Apr 29 '22 at 06:36
  • 2
    Installed rsyslogd, took out systemd-journald. /etc/rsyslog.conf helpfully had all the necessary "running without journald" configuration commented out, just waiting to be taken into use. Rock solid. – Jonas Berlin Apr 29 '22 at 06:39
  • @JonasBerlin: Can you make your comment about rsyslogd into an answer so I can upvote it? It's been years that this bug/misfeature has been in journald. – hackerb9 May 20 '23 at 17:46
  • 1
    @U.Windl *Systemd maintainer stated it's considered to be a bug in the kernel if many messages are created in a short interval.* LOLWUT?!?! What is systemd going to do in that case? Panic the kernel? "Hey, something generated a lot lot log messages! Must be something wrong with the **KERNEL**!!! Time to panic and crash everything!" Par for the systemd course, I guess... – Andrew Henle May 26 '23 at 12:47

1 Answers1

0

As syslog was neither designed to handle very large messages, nor a continuous stream of messages, you should consider writing to a plain text file instead of using syslog services.

This is specifically true if the "messages" are debug-type of messages.

Also "rate limiting" can mean two different things:

  • The number of message you want to output
  • the number of messages you can output

The manual of journald.conf explains:

If, in the time interval defined by RateLimitIntervalSec=, more messages than specified in RateLimitBurst= are logged by a service, all further messages within the interval are dropped until the interval is over.

So obviously you want to make RateLimitIntervalSec rather small (like 1 second), while making RateLimitBurst rather large (like several millions, assuming the system can handle a syslog message within less than one microsecond).

U. Windl
  • 366
  • 3
  • 17