4

I have a Python 2.7 script running on Linux that crashes with IOError: [Errno 24] Too many open files. When I run lsof -p <script_pid> to see what files the script has open, I see an increasing number of anon_inode files.

This script first downloads files from S3 using eventlet for concurrency. It then processes the downloaded files using multiprocessing.dummy for multithreading. I have run the multithreaded code in isolation and found that it only leaks file descriptors when I include the following monkey patching for eventlet:

patcher.monkey_patch(thread=False)

Any ideas on how I could resolve this would be much appreciated!

Hélène Martin
  • 1,409
  • 2
  • 15
  • 42
  • 1
    I registered your issue here https://github.com/eventlet/eventlet/issues/197 would you share your isolation code for tests? And do you experience any problems with just `monkey_patch()` ? – temoto Feb 09 '15 at 22:02
  • Thanks, @temoto! Just monkey_patch() caused serious problems with multithreading. I just discovered that I could selectively patch with import_patched -- do you know if eventlet.import_patched('boto.s3.connection') is sufficient? I guess I don't fully understand what needs to be patched. If that's not an appropriate fix, I'll post isolation code. – Hélène Martin Feb 09 '15 at 22:25
  • Selective patch generally works well. Except for when target module does import in function some time later -- it will import non-patched version then. Please, post your code anyway. – temoto Feb 11 '15 at 14:23
  • Maybe it is related to this: https://stackoverflow.com/questions/9959598/multiprocessing-and-garbage-collection. – asmaier Mar 24 '15 at 15:51

1 Answers1

1

I also run into this issue, on Ubuntu at least the open files limit for normal users defaults to 4096. So, if you are going to have more than ~4000 simultaneous connections you need to bump this up.

Solution: Raise open file descriptor limits

Here's how to do it on ubuntu

add a line to end of /etc/security/limits.conf like

* soft nofile 16384    
* hard nofile 16384

The first column describes WHO the limit is to apply for. * is a wildcard, meaning all users. To raise the limits for root, you have to explicitly enter 'root' instead of '*'.

You also need to edit /etc/pam.d/common-session* and add the following line to the end:

session required pam_limits.so

logout and then back in before you can use the new max limit, test with

ulimit -n

https://askubuntu.com/questions/162229/how-do-i-increase-the-open-files-limit-for-a-non-root-user

Also here is a good article on the caveats in using eventlets https://code.mixpanel.com/2010/10/29/gevent-the-good-the-bad-the-ugly/

Community
  • 1
  • 1
dawn360
  • 11
  • 3