Python ImageIO: Too many open files

Question

I am using imageio in python in order to open all video files in a directory and convert them to numpy arrays.

Here is the script I am using:

  1 from __future__ import print_function
  2 from avi_to_numpy import *
  3 from os import listdir
  4 import numpy as np
  5 import imageio 
  6       
  7 class_path = '../Diving/'
  8 max_frames = 16
  9 stride = 8
 10 videos = [vid for vid in listdir(class_path)]
 11 train = []
 12 
 13 for vid in videos:
 14     print(str.format('Loading {}...', vid), end="")
 15     filename = class_path + vid
 16     reader = imageio.get_reader(filename, 'ffmpeg')
 17     frames = []
 18     
 19     for i, im in enumerate(reader):
 20         if len(frames) == max_frames:
 21             break
 22         
 23         if i % stride == 0:
 24             frames.append(im)
 25     
 26     reader.close()
 27     train.append(np.array(frames))
 28     print('done')        
 29 
 30 
 31 print(len(train))

Eventually this script crashes with the following error output:

Traceback (most recent call last):
  File "load_class_test.py", line 16, in <module>
    reader = imageio.get_reader(filename, 'ffmpeg')
  File "/usr/local/lib/python2.7/site-packages/imageio/core/functions.py", line 111, in get_reader
    return format.get_reader(request)
  File "/usr/local/lib/python2.7/site-packages/imageio/core/format.py", line 158, in get_reader
    return self.Reader(self, request)
  File "/usr/local/lib/python2.7/site-packages/imageio/core/format.py", line 207, in __init__
    self._open(**self.request.kwargs.copy())
  File "/usr/local/lib/python2.7/site-packages/imageio/plugins/ffmpeg.py", line 260, in _open
    self._initialize()
  File "/usr/local/lib/python2.7/site-packages/imageio/plugins/ffmpeg.py", line 326, in _initialize
    stdout=sp.PIPE, stderr=sp.PIPE)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1223, in _execute_child
    errpipe_read, errpipe_write = self.pipe_cloexec()
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1175, in pipe_cloexec
    r, w = os.pipe()
OSError: [Errno 24] Too many open files

I am closing the Reader object from imageio. It seems as if the files opened by ffmpeg are not being closed properly.

Is there an obvious step I am missing here? Am I closing the files properly?

EDIT: Found temporary solution. Opened a new issue on github.

I was able to resolve the issue by uncommenting the following lines of code from 'imageio/plugins/ffmpeg.py':

381         def _close_streams(self):
382             for std in (self._proc.stdin,
383                         self._proc.stdout,
384                         self._proc.stderr):
385                 try:
386                     std.close()
387                 except Exception:  # pragma: no cover
388                     pass

I then added a call to the above function in _close(self):

271         def _close(self):
272             self._terminate(0.05)  # Short timeout
273             self._close_streams()
274             self._proc = None

I am not sure what the side effects of doing this are, but it provides a solution for me.

Here is the link to the issue: https://github.com/imageio/imageio/issues/145