So we have an apache server running in Linux with no issue.
Yesterday we made a server restart and since then we noticed that the number of open files by the apache processes is keeping increasing until the server after few hours failed to receive requests with below errors filling the error log:
(24)Too many open files: file permissions deny server access: /var/www/html/*X*
(24)Too many open files: /var/www/html/.htaccess pcfg_openfile: unable to check htaccess file, ensure it is readable
We check the open files and we noticed that it is really reaching the limit (which is set to 4096). We can increase the limit but this will probably increase the up time only and will not solve the issue as this behavior is not normal. No extra load is there.
And checking the open files, we noticed that the abnormal increasing open files are unix socket, that is lsof -p PID is full of:
httpd 19848 apache 13u unix 0xffff880238a63380 0t0 204055 socket
httpd 19848 apache 14u unix 0xffff8800aa3123c0 0t0 204183 socket
httpd 19848 apache 15u unix 0xffff880037a0acc0 0t0 204201 socket
I am not an experienced Linux administrator ( the administrator is not present at the moment), so I am not quit sure how to investigate the issue further.
And yes I've went through many related questions but reached no where.
Update
Testing with ss
command reveal that many sockets as below:
u_str UNCONN 0 0 * 331727 * 0
u_str UNCONN 0 0 * 331729 * 0
u_str UNCONN 0 0 * 331731 * 0
u_str UNCONN 0 0 * 331733 * 0
u_str UNCONN 0 0 * 331735 * 0