I've set some scheduled jobs on gitlab. It is running some parallel jobs every minute and reaches about 500 simultaneous process . After some point runner throws below error and fails.
Running with gitlab-runner 15.5.0 (0d4137b8)
on XXXX y4s23Bpz
Preparing the "shell" executor
00:00
Using Shell executor...
Preparing environment
00:00
ERROR: Job failed (system failure): prepare environment: failed to start process: fork/exec /usr/bin/su: too many open files. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information
I've increased open files limit but didn't resolve problem. I couldn't figure out which limit it hits.
~$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 515311
max locked memory (kbytes, -l) 65536
max memory size (kbytes, -m) unlimited
open files (-n) 999999
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 515311
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
