How to increase carbon-cache nofile limit on Debian 7?

Question

I'm running Graphite 0.9.12 on a Debian 7 server.

Recently I found errors in carbon-cache's log, saying "too many open files". According to carbon-cache's website, I need to increase nofile limit for carbon-cache. http://graphite.readthedocs.org/en/latest/carbon-daemons.html

So I increased both system wide and per-process limit:

cat /proc/sys/fs/file-max
5000000

And in /etc/security/limits.conf

* hard nofile 1000000
* soft nofile 1000000
root soft nofile 1000000
root hard nofile 1000000
www-data soft nofile 1000000
www-data hard nofile 1000000

Using ulimit command, I can confirm root/me/www-data's file limit has been increased.

I restarted carbon-cache multiple times. However, the per-process file limit for carbon-cache is not increased, still at 1024:4096

Max open files            1024                 4096                 files

I'm using /etc/init.d/carbon-cache start script to restart carbon-cache, sudo service carbon-cache restart (or stop and then start)

The script is:

#!/bin/sh
### BEGIN INIT INFO
# Provides:          carbon-cache
# Required-Start:
# Required-Stop:
# Default-Start:
# Default-Stop:
# Short-Description: initscript for runit-managed carbon-cache service
### END INIT INFO

# Author: Opscode, Inc. <cookbooks@opscode.com>

PATH=/sbin:/usr/sbin:/bin:/usr/bin
DESC="runit-managed carbon-cache"
NAME=carbon-cache
RUNIT=/usr/bin/sv
SCRIPTNAME=/etc/init.d/$NAME

# Exit if runit is not installed
[ -x $RUNIT ] || exit 0

# Load the VERBOSE setting and other rcS variables
. /lib/init/vars.sh

# Define LSB log_* functions.
# Depend on lsb-base (>= 3.0-6) to ensure that this file is present.
. /lib/lsb/init-functions


case "$1" in
  start)
        [ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC " "$NAME"
        $RUNIT start $NAME
        [ "$VERBOSE" != no ] && log_end_msg $?
        ;;
  stop)
        [ "$VERBOSE" != no ] && log_daemon_msg "Stopping $DESC" "$NAME"
        $RUNIT stop $NAME
        [ "$VERBOSE" != no ] && log_end_msg $?
        ;;
  status)
        $RUNIT status $NAME && exit 0 || exit $?
        ;;
  reload)
        [ "$VERBOSE" != no ] && log_daemon_msg "Reloading $DESC" "$NAME"
        $RUNIT reload $NAME
        [ "$VERBOSE" != no ] && log_end_msg $?
        ;;
  force-reload) ................ SO ON

Does anyone know how can I make carbon-cache apply my new nofile limit? Should I modify the start script or do something else?

===============================================================

Update:

I found another file may be related in my system. /etc/sv/carbon-cache/run.

Originally, the content shows as:

#!/bin/sh
exec 2>&1
exec chpst -u www-data -l /opt/graphite/storage/carbon-cache.lock -- /opt/graphite/bin/carbon-cache.py --pid /opt/graphite/storage/carbon-cache-a.pid --debug start

I edited the file to:

#!/bin/sh
ulimit -n 999999
exec 2>&1
exec chpst -o 999999 -u www-data -l /opt/graphite/storage/carbon-cache.lock -- /opt/graphite/bin/carbon-cache.py --pid /opt/graphite/storage/carbon-cache-a.pid --debug start

Now in /proc/pid/limits, both soft and hard limit are 999999.

kwarunek · Answer 1 · 2016-01-05T06:51:29.667

0

With the ulimit command, you can change your soft limits for the current shell environment, and hard limits if running as root.

Scripts in init.d start with new shell env (session) the ulimit must be applied once again. You could add to /etc/init.d/carbon-cache before case statement something like:

ulimit -n 999999

This will set for current (init.d start) session new limit every start/restart/reload/whatever. Limits are inherited from a parent process to its child processes, so carbon-cache will be affected as well.

edit

set the hardlimit with ulimit -H
is it supervisor? if so, also add minfs in /etc/supervisor/supervisord.conf in section [supervisord].

minfds=10000

edited Jan 05 '16 at 06:51

answered Jan 04 '16 at 23:51

kwarunek

12,141
4
43
48

I added ulimit -n 10000 to start script. However when I'm checking new pid's limits, (/proc/pid/limits) it's still showing as 1024/4096, but not 10000. And ideas? – Liu Yunao Jan 05 '16 at 02:24
try to increase hardlimit with `ulimit -H 10000` – kwarunek Jan 05 '16 at 06:37
Using ulimit -H didn't help as well. However, as I looked into some other possibly related files, I found this one "/etc/sv/carbon-cache/run". Inside the "run" file, it shows "exec chpst -u www-data -l /opt/graphite/storage/carbon-cache.lock -- /opt/graphite/bin/carbon-cache.py --pid /opt/graphite/storage/carbon-cache-a.pid --debug start" So I searched command chpst, and found a -o option to set file limit. I added -o 99999 to "run" file, and restarted carbon-cache. Now both soft-limit and hard-limit are 4096. It seems the -o option raised soft limit, but not hard limit. Any way to fix this? – Liu Yunao Jan 05 '16 at 21:52
you wrote that you edited /etc/security/limits.conf. have you rebooted the system? – kwarunek Jan 05 '16 at 22:10
and `chpst` can only change process (not session) conf/state within hard limits. It is impossible to change hard limits with it. – kwarunek Jan 05 '16 at 22:20

How to increase carbon-cache nofile limit on Debian 7?

1 Answers1