6

I have an AWS EC2 instance running a CentOS 7.7.1908 (systemd 219) with a server application. The server logs quite a lot of information to the system logs (using syslog).

I have recently enabled persistent storage of the system logs using this answer. Since then, the memory consumption of systemd-journald is constantly growing.

After a full day, systemd-journald ends up using more than 250M of RAM.

I did a quick test to confirm that the journald RAM usage is actually growing.

Test :

  • Get the systemd-journald memory usage
  • Log a bunch of data to the system journal using logger
  • Get the systemd-journald memory usage
  • Flush the in-memory logs (if any) to /var/log/journal using journalctl --flush
  • Restart the systemd-journald service
  • Get the systemd-journald memory usage

The results :

# ps aux | grep journald | grep -v grep
root     23963  0.0  0.1  61320  2500 ?        Ss   15:03   0:00 /usr/lib/systemd/systemd-journald
# for i in {0..7000}; do logger -t TEST -p err `python -c 'print "A"*1000'`; done;
# ps aux | grep journald | grep -v grep
root     23963  0.1  0.2  69512 11964 ?        Ss   15:08   0:00 /usr/lib/systemd/systemd-journald
# journalctl --flush
# ps aux | grep journald | grep -v grep
root     23963  0.1  0.2  69512 11964 ?        Ss   15:08   0:00 /usr/lib/systemd/systemd-journald
# systemctl restart systemd-journald
# ps aux | grep journald | grep -v grep
root     24237  0.0  0.1  55416  2492 ?        Ss   15:08   0:00 /usr/lib/systemd/systemd-journald

The memory usage goes from 2,5M to almost 12M after logging rougly 7M of data. Restarting the daemon brings the memory usage back to ~2.5M.

My interpretation :

  • systemd-journald memory usage is proportional to the amount of data it has processed.
  • The parameter RuntimeMaxUse does not limit this memory usage.
  • Restarting systemd-journald clears a buffer, that I can not configure the size of.
  • There is no configurable limit to the amount of memory systemd-journald may use.
  • The memory use is not due to logs waiting to be flushed to the disk.

My journald.conf :

[Journal]
#Storage=auto # /var/log/journal is created and used, so this is equivalent to 'Persistent'
SyncIntervalSec=1m
RateLimitInterval=0
RateLimitBurst=0
SystemMaxUse=4G
RuntimeMaxUse=1M

How can I solve this "leak" without restarting the service every day?

MikaDo-
  • 61
  • 1
  • 3

0 Answers0