systemctl limits - solr complaining

Question

I'm building a solr server (on Ubuntu 18.04, using the repo solr-common and solr-jetty). On startup, solr was reporting that nfile and nproc (1024, 6721 resp) was set too low. I ran systemctl edit solr and created an override as follows:

[Service]
LimitNOFILE=65000
LimitNPROC=65000

I then restarted the service - solr still reporting the same issue.

I added /etc/security/limits.d/solr containing:

solr hard nofile 65535
solr soft nofile 65535
solr hard nproc 65535
solr soft nproc 65535

It is still reporting the same issue after restarting the service:

# systemctl status solr
● solr.service - LSB: Controls Apache Solr as a Service
   Loaded: loaded (/etc/init.d/solr; generated)
  Drop-In: /etc/systemd/system/solr.service.d
           └─override.conf
   Active: active (exited) since Mon 2020-03-30 14:55:49 BST; 6s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 6848 ExecStop=/etc/init.d/solr stop (code=exited, status=0/SUCCESS)
  Process: 6973 ExecStart=/etc/init.d/solr start (code=exited, status=0/SUCCESS)

Mar 30 14:55:43 dev-a01-si-solr.bip solr[6973]: *** [WARN] *** Your open file limit is currently 1024.
Mar 30 14:55:43 dev-a01-si-solr.bip solr[6973]:  It should be set to 65000 to avoid operational disruption.
Mar 30 14:55:43 dev-a01-si-solr.bip solr[6973]:  If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
Mar 30 14:55:43 dev-a01-si-solr.bip solr[6973]: *** [WARN] ***  Your Max Processes Limit is currently 6721.
Mar 30 14:55:43 dev-a01-si-solr.bip solr[6973]:  It should be set to 65000 to avoid operational disruption.
Mar 30 14:55:43 dev-a01-si-solr.bip solr[6973]:  If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
Mar 30 14:55:49 dev-a01-si-solr.bip solr[6973]: [194B blob data]
Mar 30 14:55:49 dev-a01-si-solr.bip solr[6973]: Started Solr server on port 8983 (pid=7045). Happy searching!
Mar 30 14:55:49 dev-a01-si-solr.bip solr[6973]: [14B blob data]
Mar 30 14:55:49 dev-a01-si-solr.bip systemd[1]: Started LSB: Controls Apache Solr as a Service.

What am I doing wrong here?

update After amending /etc/systemd/system.conf to contain...

DefaultLimitNOFILE=65000
DefaultLimitNPROC=65000

Solr is no longer complaining about the file limit but still complaining about the process limit. WTF Pottering?

  Drop-In: /etc/systemd/system/solr.service.d
           └─override.conf
   Active: active (exited) since Mon 2020-03-30 15:21:59 BST; 14s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 1141 ExecStart=/etc/init.d/solr start (code=exited, status=0/SUCCESS)

Mar 30 15:21:51 dev-a01-si-solr.bip solr[1141]:  If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
Mar 30 15:21:51 dev-a01-si-solr.bip solr[1141]: *** [WARN] ***  Your Max Processes Limit is currently 6721.
Mar 30 15:21:51 dev-a01-si-solr.bip solr[1141]:  It should be set to 65000 to avoid operational disruption.
Mar 30 15:21:51 dev-a01-si-solr.bip solr[1141]:  If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
Mar 30 15:21:51 dev-a01-si-solr.bip solr[1141]: Warning: Available entropy is low. As a result, use of the UUIDField, SSL, or any other features that require
Mar 30 15:21:51 dev-a01-si-solr.bip solr[1141]: RNG might not work properly. To check for the amount of available entropy, use 'cat /proc/sys/kernel/random/entropy_avail'.
Mar 30 15:21:59 dev-a01-si-solr.bip solr[1141]: [230B blob data]
Mar 30 15:21:59 dev-a01-si-solr.bip solr[1141]: Started Solr server on port 8983 (pid=1459). Happy searching!
Mar 30 15:21:59 dev-a01-si-solr.bip solr[1141]: [14B blob data]
Mar 30 15:21:59 dev-a01-si-solr.bip systemd[1]: Started LSB: Controls Apache Solr as a Service.

Amending user.conf to match did not help.

Update 2 Well, this just keeps getting better and better. The disappearance of the nfile warning came after a reboot of the host. When I subsequently run systemctl restart solr I get this:

Mar 30 15:39:21 dev-a01-si-solr.bip solr[2503]: *** [WARN] *** Your open file limit is currently 1024.
Mar 30 15:39:21 dev-a01-si-solr.bip solr[2503]:  It should be set to 65000 to avoid operational disruption.

FFS!

Now, where did I put that Centos 5 CD?

Update 3

It turns out that this was no longer the packaged solr. Unbenknownst to me, someone had problems getting the original build to work and found a tutorial on the internet on how to install from tarball. So I now have a system with half tarball/half repo solr which we can't patch / upgrade.

score 1 · Accepted Answer · answered Mar 30 '20 at 15:18

On Ubuntu 18.04, solr is actually started by an initV script invoked from systemd.

I've got rid of the warnings by

1) removing /etc/security/limits.d/solr and adding the following in /etc/security/limits.conf:

*                hard   nofile 2048
*                hard   nproc  2048

root             hard   nofile unlimited
root             soft   nofile unlimited
root             hard   nproc  unlimited
root             soft   nproc  unlimited

solr hard nofile 65535
solr soft nofile 65535
solr hard nproc 65535
solr soft nproc 65535

2) Adding this to the top of /etc/init.d/solr

ulimit -n 65000
ulimit -u 65000

(although the -u line doesn't appear to be doing anything)

3) running systemctl daemon-reload

4) doing a hard reboot

And the warnings stay away when I run systemctl reload solr.

But I now have bots of configuration scattered all over the place and I need to trim some of these out.

systemctl limits - solr complaining

1 Answers1