Our django celery server is running with concurrency value of 500 with eventlet as execution pool. Recently, we encountered this issue and restarting the server fixed it. I am attaching the traceback of the error.
python package versions: Django 4.1.7 celery 5.2.3 eventlet 0.33.3
I checked the soft and hard limits of the open files if these are of any use.
soft limits:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 31071
max locked memory (kbytes, -l) 65536
max memory size (kbytes, -m) unlimited
open files (-n) 2048
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 31071
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
hard limits:
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 31071
max locked memory (kbytes, -l) 65536
max memory size (kbytes, -m) unlimited
open files (-n) 16384
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) unlimited
cpu time (seconds, -t) unlimited
max user processes (-u) 31071
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
I have been using eventlet for the past 6 or 7 months and this is the first time we have faced this issue. how do we prevent this from happening again
Traceback (most recent call last):
File "/home/ubuntu/project_data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 476, in fire_timers
timer()
File "/home/ubuntu/project_data/venv/lib/python3.10/site-packages/eventlet/hubs/timer.py", line 59, in __call__
cb(*args, **kw)
File "/home/ubuntu/project_data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 224, in main
self._resolve_links()
File "/home/ubuntu/project_data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 240, in _resolve_links
f(self, *ca, **ckw)
File "/home/ubuntu/project_data/venv/lib/python3.10/site-packages/celery/concurrency/eventlet.py", line 62, in _entry_exit
g.wait()
File "/home/ubuntu/project_data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 181, in wait
return self._exit_event.wait()
File "/home/ubuntu/project_data/venv/lib/python3.10/site-packages/eventlet/event.py", line 132, in wait
current.throw(*self._exc)
File "/home/ubuntu/project_data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 221, in main
result = function(*args, **kwargs)
File "/home/ubuntu/project_data/venv/lib/python3.10/site-packages/kombu/asynchronous/timer.py", line 70, in __call__
return self.fun(*self.args, **self.kwargs)
File "/home/ubuntu/project_data/venv/lib/python3.10/site-packages/kombu/asynchronous/timer.py", line 137, in _reschedules
return fun(*args, **kwargs)
File "/home/ubuntu/project_data/venv/lib/python3.10/site-packages/celery/worker/heartbeat.py", line 45, in _send
loadavg=load_average(),
File "/home/ubuntu/project_data/venv/lib/python3.10/site-packages/celery/utils/sysinfo.py", line 23, in load_average
return _load_average()
File "/home/ubuntu/project_data/venv/lib/python3.10/site-packages/celery/utils/sysinfo.py", line 13, in _load_average
return tuple(ceil(l * 1e2) / 1e2 for l in os.getloadavg())
OSError: Load averages are unobtainable
