pacemaker.log file is flooding with “Too many open files (24)”

Question

My system is currently at Centos 8 and my /var/log/pacemaker/pacemaker.log file is flooding with GB of data and most of the error messages shows the below message. These messages are more than 1000 in numbers and they are filling up like almost 10-15 GB of data in a matter of 10 minutes

Oct 13 12:10:46 vm01 pacemaker-attrd     [23173] (qb_ipcs_us_connection_acceptor)   error: Could not accept client connection: Too many open files in system (23)

Oct 13 12:10:46 vm01 pacemaker-attrd     [23173] (qb_ipcs_us_connection_acceptor)   error: Could not accept client connection: Too many open files in system (23)

Earlier in my environment I was using Centos 7 and I never saw this log flooding. After upgrading to Centos 8 this issue started

Below is the ulimit output from Centos 8

[root@vm01 ~]# ulimit -n
65536

Problem is that this issue occurs randomly at a given day and I always have to manually empty the /var/log/pacemaker/pacemaker.log files in order to ensure that system doesn't halt after reaching 100% storage

Output of lsof when this issue occured.

 [root@vm01 ~]# lsof | wc -l
    416113

I've been told to modify the following lines on /etc/security/limits.conf, which is to change the 65536 to 524288

[root@vm01 ~]# cat /etc/security/limits.conf  | grep root
root hard nofile 1048576
root soft nofile 65536

One more change is to make the following change on /etc/sysctl.conf , which is to again change 65536 to 524288, followed by runing /sbin/sysctl -p

[root@vm01 ~]# cat /etc/sysctl.conf | grep -i fs.file
fs.file-max = 65536

Kindly assist me on this problem and also let me know if the change for sysctl.conf and limits.conf can be applied on the environment to fix this

Answer 1

What is the ulimit of your pacemaker-process?

Find out with cat /proc/<procid>/limits

example:

$ cat /proc/8933/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             128443               128443               processes
Max open files            16384                16384                files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       128443               128443               signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

This will help you to analyze this further.

If your pacemaker is running with a lower ulimit and you're using systemd, you can just add a systemd-drop-in (configuration) to add configuration to your service without manipulating the original unit-file for pacemaker.

If your systemd-unit-file has this path: /etc/systemd/system/pacemaker.service, you can create a directory /etc/systemd/system/pacemaker and put a file with the extension .conf. into this directory.

The content of this file could be:

[Service]
LimitNOFILE=65536

Do systemctl daemon-reload afterwards, restart pacemaker and the changes should get applied.

Reference: https://www.freedesktop.org/software/systemd/man/systemd.exec.html