3

I have a multithread linux application running on a linux system

The application has been working successfully in different Linux systems and kernels without never noticing this issue.

We are currenlty using this kernel

#ulimits -a
Linux AM38 4.9.0-8-rt-amd64 #1 SMP PREEMPT RT Debian 4.9.130-2 (2018-10-27) x86_64 GNU/Linux

We have been using this kernel for 1 year now without problems.

The application can have some clients that connects externally. When a client connects a couple of threads per client get created.

Recently I've run into a problem that pthread_create return EAGAIn. I managed to design a stress test that reproduces the failure. It takes 2 hours to reproduce. A similar amount of time that took to fail in production.

Once I was able to reproduce the problem, I have gone back to versions used in production without problems but the problem now arises on the old versions too. So I think we always had the issue but we have now an user case that highlights the problem.

Basically the test simulates cuts in communications for 30 seconds, so all the clients gets disconnected, and then I let the system works normally for another 30 seconds, for the clients to reconnect. I added a 450 ms latency to make the stress even further when trying to reconnect. There are 30 clients only.

In production and in my stress lab conditions the problem appears after 2h of start stressing the system.

I've checked for zombies to be sure that I was joining the threads properly. htop or ps never show any thread as Z or defunct.

I have monitored the system with htop and I never saw more than 46 tasks and 140 threads total in the system.

I've checked system limits and look ok.

# ulimit -a
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31414
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 31414
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

# cat /proc/sys/kernel/pid_max
327687
# cat /proc/sys/kernel/threads-max
62828
# free -h
              total        used        free      shared  buff/cache   available
Mem:           7.7G        470M        3.7G         83M        3.5G        7.0G
Swap:            0B          0B          0B

If I do a ps it looks like

# ps -axH | grep myapplication
 3910 tty5     Sl+    4:23 myapplication -v
 3910 tty5     Sl+    1:41 myapplication -v
 3910 tty5     Sl+    0:02 myapplication -v
 3910 tty5     Sl+    0:00 myapplication -v
 3910 tty5     Sl+    0:46 myapplication -v

.... same looking lines here

 3910 tty5     Sl+    0:00 myapplication -v
 3910 tty5     Sl+    0:47 myapplication -v
 3910 tty5     Sl+    0:00 myapplication -v
 3910 tty5     Sl+    0:48 myapplication -v
 3910 tty5     Sl+    0:00 myapplication -v
 3910 tty5     Sl+    0:49 myapplication -v
 3910 tty5     Sl+    0:51 myapplication -v

total threads: 134

I can connect to the system and execute programs and the web server on the system runs. Only that process seems to fail.

If I stop/start that process all goes back to normal for another 2h.

Here pthread_create fails with EAGAIN I've found that I might be hitting this bug

https://bugzilla.kernel.org/show_bug.cgi?id=154011

But I don't know how to confirm it and what to do to solve my issue. It does not seems it is fixed.

Suggestions?

caf
  • 233,326
  • 40
  • 323
  • 462
webadm
  • 51
  • 5
  • 1
    I would strongly suggest you just create all the threads you need at startup and don't create and destroy threads as the program is running. – David Schwartz Aug 11 '19 at 03:56
  • Was there any change to the kernel or to the global scheduling policy (i.e. something with `SCHED_DEADLINE`)? Have you tried applying the kernel patch proposed on Bugzilla and see if it changes anything? – root Aug 11 '19 at 05:14
  • 2
    Bug 154011 is about sporadic `EAGAIN` when the process is just below the `NPROC` limit. That doesn't match your symptoms. Creating a https://stackoverflow.com/help/minimal-reproducible-example will go a long way to help you. – Employed Russian Aug 12 '19 at 06:00
  • David Schwartz - Use the a thread pool was in my list but it is a big long refactoring. and my last option. I want to understand first what it is happening because the current design has been working the way it is for years. – webadm Aug 12 '19 at 14:05
  • root - yes...there was a change on the priority of 1 task and it changed to use a RT scheduler. But that thread gets created at the beginning and runs continuously. The thread that get created/destroyed on the stress test are under default Linux scheduler. – webadm Aug 12 '19 at 14:09
  • Employed Russian - I do agree on the min-example but not totally sure how to reproduce the conditions yet in order to achieve that. It would be really useful when I achieve to have the system under the failure conditions to query linuxabash to see why the process is reporting EAGAIN on all attempts to create a thread. Any ideas where to start looking and what? – webadm Aug 12 '19 at 14:16
  • You seem to appreciate that the issue manifests only when the system is under stress, and only sporadically even then. If, then, you're indeed looking for an answer that explains the nature of the problem, then we need to understand specifically what the program is doing, in the form of an [mre]. – John Bollinger Aug 12 '19 at 14:34
  • On the other hand, if you're just looking for a mitigation then since you seem only transiently to be hitting a system limit, you could consider doing exactly what `EAGAIN` is mnemonic for: try again. Putting in a short delay before the retry will make it more likely that whatever issue you're running into has a chance to sort itself out before the next try. If you like, you can look at that as dynamically reducing the stress on the system when necessary by spreading out thread-startup work over a longer time. – John Bollinger Aug 12 '19 at 14:37
  • John Bollinger -- it is not soporadically. Once pthread_created fails, all posterior calls to pthread_create fail too. I have serveral clinet -server protocols supported, with different implementation. And all of them failed to create new threads when the error manifets the first time. It looks like I have reached a limit of some kind, but I can figure it out which one. The system is continously trying AGAIN and failing ecah time. It would be helpful to know what things under /proc can be checked to see what limit I have reached. – webadm Aug 12 '19 at 15:03
  • I will recollect what I know and try to isolate the code in a minimal sample to see it reproduces. – webadm Aug 12 '19 at 15:05
  • 1
    If the stress test fails consistently in about the same amount of time, and the application never recovers from it unless shut down and restarted, then that suggests you are exhausting a system resource. That does not appear to be memory, and probably not process or thread slots, so perhaps it is open files? – John Bollinger Aug 12 '19 at 16:38
  • John Bollinger - I agree with you. It is what it looks like. The threads that get created/destroyed on the stress test does not manage any files, but sockets. The application does not open/close files periodically either., I've tested netstat looking for too many sockets but it is all normal too. 20-25 entries that are the amount of clients connecting/disconnecting plus the normal services. How could number of files/sockets exhaustion could affect pthread_create? Also how could query the resources the application is using when I have the problem reproduced? – webadm Aug 12 '19 at 18:47
  • I'm now a testing minimize-reproducible-example. and I'm playing with it. In a test I'm not joining the threads in purpose. I was expecting to see zoombies threads with "ps -asH". But I'm not seeing them. The Thread Attributes parameter is NULL in pthread_create so I'm creating joinable threads. If ps is not showing the thread zoombies, what it is the linux command that shows the thread zoombies? – webadm Aug 13 '19 at 00:14
  • Sockets *are* files in the relevant sense of the latter term. I'm uncertain what exact mechanism might lead to open-file exhaustion causing `pthread_create` to fail, but it's not out of the question that there is a need to at least transiently open a new file. My go-to for checking open files would be the `lsof` command. Do use the `-p ` option to narrow the listing to just the process of interest. – John Bollinger Aug 13 '19 at 00:21

1 Answers1

0

I've found what it is.

I'm not calling a join_thread in one place, but I was expecting to see the threads as zoombies with ps, This issue stackoverflow.com/questions/31765867/… gave me the place to where to look at.

The best hint I've found to confirm the problem is the virtual memory usage. In my case, the non joined threads were increasing the amount of reserved virtual memory. My Application is a 32bit one so once we achieved 4GB of virtual memory it was game over. Thread Virtual memory usage can be easily see just executing htop.

The link I've found proposes a technique that gives you the answer to where the leaks are if you can modify the sources and you are using the pthread funcions directly.

The Virtual memory increases is an indirect data that can rise a flag about the reason of the failure in production.

Thanks everyone for the help.

webadm
  • 51
  • 5