3

There is a known issue that uuid.uuid1() function creates a file descriptor leak under multi threaded environments. This can cause servers to crash after reaching the limit of max number of open files. Recently, our production servers are facing this issue and we have to restart them everyday. I am trying to replicate this fd leak through a sample code. However, I am unable to do so. Here is the sample code:

import threading
import uuid
import time

class uuidGenerator(threading.Thread):
    """multithreaded program for trying to replicate the fd leak problem in uuid.uuid1() under large number of threads"""

    def __init__(self):
        threading.Thread.__init__(self)

    def run(self):
        self.generate_uuid();

    def generate_uuid(self):
        print uuid.uuid1()
        print "\n"
        time.sleep(1)

if __name__ == "__main__" :
    threads=[]
    try:
        for i in range(0,100000):
            thread=uuidGenerator()
            thread.start()
            threads.append(thread)
    except:
        print "Error: unable to start thread"

    print len(threads)
    for thread in threads:
        thread.join()
    while 1:
        pass

How can I reproduce this issue, and how can I fix it?

My python version is 2.7.10

  • If you're just out to get a simulated version of `to many file descriptors open`, why not just do `open_files = []; open_files.append(open('/tmp/dummy0', 'w'))`? or use `os.fdopen`? – Torxed Apr 13 '16 at 06:16
  • Well, I can do that. But i am curious to know how does `uuid.uuid1()` is not creating a fd leak by this code and i have to find a work around for this leak as well :) – Rajat Srivastava Apr 13 '16 at 06:20
  • I'm not entirely sure this is reproducible, `/var/lib/libuuid/clock.txt` is never opened in Python3 for me. Either this is monkey-patched after Mar 23, 2015 by Python before packaging the binary. Or the `libuuid` team has patched it since then. – Torxed Apr 13 '16 at 06:55
  • Are you sure you're getting a `fd` error, and not `RuntimeError: can't start new thread`? – Torxed Apr 13 '16 at 06:56
  • Yes i checked on the production server with `sudo ls -l /proc//fd` i am seeing lots of `/var/lib/libuuid/clock.txt` in it. but this code doesn't show such fd and i am not getting runtime error because my test server can support this number of threads. – Rajat Srivastava Apr 13 '16 at 07:04
  • Trying to reproduce this on `2.7.11` doesn't work. I have zero files opening on anything called `*clock*` in my production environment. I'm running this code for comparison: https://gist.github.com/Torxed/8cbe5ac75e7098f13ed73949acfeec9c – Torxed Apr 13 '16 at 07:17
  • @Torxed..yes this code does not reproduce the error and my question is why is this so?? I do not think the bug is fixed because one of our servers is having this leakage – Rajat Srivastava Apr 13 '16 at 08:29
  • I'm sure someone more qualified can find the issue in a heart beat, but until then I'd suggest you do `python -m trace --trace your_code.py > trace_script.log` and check the output for `clock.txt` and step around the code from there. Especially see which function calls creates the file and isolate the function call as a primary suspect and see if you can reproduce it in your miniature script. – Torxed Apr 13 '16 at 08:52
  • A guess: you may be falling back to python code - do you have both ctypes and libuuid installed? (https://github.com/python/cpython/blob/2.7/Lib/uuid.py#L442) And if you `strace` you app, can you see `libuuid` being successfully loaded? – viraptor Apr 13 '16 at 08:56
  • Any solution for this ? I am hiting this issue now :( – Nilesh May 04 '17 at 02:50
  • @Nilesh..you can use `uuid4()` for generating random uuids..since it does not use `/var/libuuid/clock.txt` file so there will not be any fd leaks and collision rate should be negligibly low. – Rajat Srivastava May 04 '17 at 09:30
  • @RajatSrivastava I am using `uuid1` in cassandra's `TimeUUID`, so I dont have option to use `uuid4` :(. My detail question is http://stackoverflow.com/questions/43807341/cassandra-timeuuid-need-to-replace-due-to-uuid-file-descriptor-issue – Nilesh May 05 '17 at 19:06

0 Answers0