8

I've searched for a few days regarding this issue but have come to no solution. I have a big script (I'm trying to concatenate large number of videos, ~100-500), which is why I was getting the error "Too many files open". Reading Zulko's responses to other issues, I saw that it was necessary to delete each VideoFileClip instance manually like this:

del clip.reader
del clip

The issue I'm facing is that I have a simple hello_world trying to do this, but get the error VideoFileClip instance has no attribute 'reader' Here's the code:

from moviepy.editor import *
rel_path = "main.py"
file_path="hello_world.mp4"
newVideo = VideoFileClip(file_path)
del newVideo.reader
del newVideo

I'm using El Capitan (OS X), have updated MoviePy, Numpy, ImageMagick, and all packages I've seen as required, but I'm still getting this error. The problem is my computer ends up freezing sometimes because it's using so much memory. I'm currently concatenating chunks of 25 videos, and trying to delete all 25 "opened files", concatenate the next 25, and so forth. After that I'd concatenate the longer videos.

Please note that without the line del newVideo.reader I still get the error Too many files open

When I try running the real script, I get the following error if I don't add newVideo.reader

Traceback (most recent call last):
  File "/Users/johnpeebles/mispistachos/vines/video/reader.py", line 135, in compile_videos
    newVideo = VideoFileClip(videoPath).resize(height=finalHeight,width=finalWidth).set_position('center').set_start(currentDuration)
  File "/Library/Python/2.7/site-packages/moviepy/video/io/VideoFileClip.py", line 55, in __init__
    reader = FFMPEG_VideoReader(filename, pix_fmt=pix_fmt)
  File "/Library/Python/2.7/site-packages/moviepy/video/io/ffmpeg_reader.py", line 32, in __init__
    infos = ffmpeg_parse_infos(filename, print_infos, check_duration)
  File "/Library/Python/2.7/site-packages/moviepy/video/io/ffmpeg_reader.py", line 237, in ffmpeg_parse_infos
    proc = sp.Popen(cmd, **popen_params)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1223, in _execute_child
    errpipe_read, errpipe_write = self.pipe_cloexec()
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1175, in pipe_cloexec
    r, w = os.pipe()
OSError: [Errno 24] Too many open files
Error compiling videos
Exception AttributeError: "VideoFileClip instance has no attribute 'reader'" in <bound method VideoFileClip.__del__ of <moviepy.video.io.VideoFileClip.VideoFileClip instance at 0x136e46908>> ignore

As requested by Tynn, I'm posting the "real code". What I'm doing here is, as I explained above, compile videos in chunks of 25 videos, then compile all these pre-compiled videos into one big video. I'm getting the error Too Many Files Open right now (if I don't add del clip.reader)

    for videoObject in data["videos"]:
        counter+=1
        #Download video and thumbnail
        downloader=urllib.URLopener()
        videoId = videoObject[API_VIDEO_ID]
        videoUrl = str(videoObject[API_VIDEO_URL])
        videoPath =os.path.join(file_dir, "tmp",str(videoObject[API_VIDEO_ID].replace("/",""))+'_video.mp4')
        thumbPath =os.path.join(file_dir, "tmp",str(videoObject[API_VIDEO_ID].replace("/",""))+'_thumb.jpg')
        currentVideoDimension = videoObject[API_VIDEO_DIMENSIONS]
        currentVideoWidth = currentVideoDimension[0]
        currentVideoHeight = currentVideoDimension[1]

        thumbUrl = str(videoObject[API_THUMB_URL])
        finalWidth = w*1.0
        finalHeight = h*1.0
        videoProportion = (float(currentVideoWidth)/float(currentVideoHeight))
        if currentVideoWidth >= currentVideoHeight:
            finalHeight = finalWidth/videoProportion
        else:
            finalWidth = finalHeight*videoProportion


        try:
            download(videoUrl, videoPath)
            download(thumbUrl, thumbPath)
        except Exception as e:
            print("Exception: "+str(e))
            print("Video ID: "+str(videoId))
            traceback.print_exc()
            continue
        #Create new video and update video duration's offset
        newVideo = VideoFileClip(videoPath).resize(height=finalHeight,width=finalWidth).set_position('center').set_start(currentDuration)
        #If it's not squared we append a video first
        if videoProportion != float(1):
            backgroundClip = ColorClip(size=((w,h)), col=colors.hex_to_rgb("#000")).set_position("center").set_start(currentDuration).set_duration(newVideo.duration)
            videos_and_subtitles.append(backgroundClip)
        #Append new video to videos

        videos_and_subtitles.append(newVideo)
        #Append subtitle to Subtitles
 #            newSubtitleText = max_text(videoObject[API_NAME],videoObject[API_AUTHOR])+" \n\n"+videoObject[API_AUTHOR]
        videoName = clean(videoObject[API_NAME])
        videoAuthor = clean(videoObject[API_AUTHOR])
        newSubtitleText = clean(max_text(videoName,videoAuthor)+" \n\n"+videoObject[API_AUTHOR])
        newSubtitle = ( TextClip(newSubtitleText,fontsize=70,color='white',font='Helvetica-Narrow',align='center',method='caption',size=titleDimensions).set_start(currentDuration).set_position((videoOffset+w,0)).set_duration(newVideo.duration) )
        videos_and_subtitles.append(newSubtitle)




        currentDuration+=newVideo.duration

        #Preprocess videos
        if counter%50==0 or len(data["videos"])==(counter):
            if closure_video_path != None and closure_video_path != "" and len(data["videos"])==(counter):
                newVideo = VideoFileClip(closure_video_path).resize(height=finalHeight,width=finalWidth).set_position((videoOffset,titleOffset)).set_start(currentDuration)
                videos_and_subtitles.append(newVideo)
                currentDuration+=closure_video_duration

            currentFilename=os.path.join(file_dir, "tmp",str(videoNumber)+fileName) 
            result = CompositeVideoClip(videos_and_subtitles,size=movieDimensions,bg_color=colors.hex_to_rgb(background_color)).set_duration(currentDuration).write_videofile(filename=currentFilename,preset='ultrafast',fps=24)

            del result

            preprocessedVideos.append(VideoFileClip(currentFilename))

            #Close files
            #close_files(videos_and_subtitles)
            for clip in videos_and_subtitles:
                try:
                    if not (isinstance(clip,ImageClip) or isinstance(clip,TextClip)):
                        del clip
                    else:
                        del clip
                except Exception,e:
                    print "Exception: "+str(e)
            #End Close files                        


            videos_and_subtitles = []
            videos_and_subtitles.append(left_holder)
            currentDuration = 0
            videoNumber+=1
            if (videoObject==data["videos"][-1]):
                break
        print("Next video")


    print("Compiling video")

    filepath = os.path.join(file_dir, "tmp",fileName) 
    result = concatenate_videoclips(preprocessedVideos).write_videofile(filename=filepath, preset='ultrafast')
    #result = CompositeVideoClip(videos_and_subtitles,size=movieDimensions,bg_color=(0,164,119)).set_duration(currentDuration).write_videofile(filename=directory+"/"+fileName,preset='ultrafast')
    print("Video Compiled")
    now = datetime.datetime.now()
    print("Finished at: "+str(now))
    return filepath
except Exception as e:
    print("Exception: "+str(e))
    print("Video ID: "+str(videoId))
    traceback.print_exc()
    rollbar.report_exc_info()
    return None
Waclock
  • 1,686
  • 19
  • 37
  • In the for loop you delete a reference you just created before. You're not closing anything there. `videos_and_subtitles = []` deletes the references in the list. Maybe even the ~50 references in your list and the content of `preprocessedVideos` are to much. – tynn Feb 02 '16 at 21:04
  • What do you mean I'm not closing anything? I delete each element in videos_and_subtitles every 50 videos (or when i get to the last video). The preprocessedVideos doesn't contain more than 1-3 elements (it's the video that was generated after compiling the 50 videos, so if I have 150 videos, i'd have 3 compositevideoclips) – Waclock Feb 02 '16 at 21:11
  • After using the CompositeVideoClip, I delete it and create a new reference to the new video (made of 50 videos). – Waclock Feb 02 '16 at 21:27
  • The way you're using `del clip` doesn't destroy any of the clip objects and thus doesn't close the files. This might be done with resetting the list. What's the value of `counter` just before the `Exception` is raised? – tynn Feb 02 '16 at 21:35
  • Why is it wrong the way im using del clip? How should I use it then :S. The value varies, sometimes it's 50, sometimes it's even bigger. How would you suggest I delete the clips? (Or why is it wrong how I do it) – Waclock Feb 02 '16 at 22:14
  • It should be done with `videos_and_subtitles = []` if `videos_and_subtitles` holds the only reference to the clip. – tynn Feb 02 '16 at 22:22
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/102478/discussion-between-waclock-and-tynn). – Waclock Feb 03 '16 at 13:14

2 Answers2

4

Zulko himself writes:

In the next versions of MoviePy just del clip will suffice.

This was before version 0.2.2 was released. So it seems like you don't need to do del clip.reader.

To be more precise you mustn't do it! VideoFileClip defines a destructor doing this for you:

def __del__(self):
  """ Close/delete the internal reader. """
  del self.reader

But since you've deleted it already you get the AttributeError:

VideoFileClip instance has no attribute 'reader'

in <bound method VideoFileClip.__del__ of <...VideoFileClip instance at 0x13..>>

tynn
  • 38,113
  • 8
  • 108
  • 143
  • Hello Tynn, thanks for your response. You're right, I forgot to mention that if I only put "del clip", I get the error 'Too Many Files Open', so it's no good either :/ – Waclock Feb 02 '16 at 13:37
  • @Waclock you need to delete all references to `clip` itself. Have you tried to do `clip.reader = None`? This should have the same effect as deleting it without the side-effect. – tynn Feb 02 '16 at 15:39
  • I'll give it a try. I believe I have, I create an array and push each clip object, then, iterate through the object and delete each instance of clip I have. – Waclock Feb 02 '16 at 16:46
  • @Waclock could you post more of your code. For me it just sounds like you're keeping quite many clip objects with this method. – tynn Feb 02 '16 at 16:58
  • You might be right. I've added the "real code", if you don't understand something, let me know. As far as I've checked, I'm deleting each instance correctly. – Waclock Feb 02 '16 at 19:30
  • Just as a heads up, this didn't really fix my problem, as references here: https://github.com/Zulko/moviepy/issues/255#issuecomment-280875615 , this is an issue. – Waclock Feb 21 '17 at 15:36
1

I have solved my question, which is described in detail at https://github.com/Zulko/moviepy.

Once you have installed ImageMagick, it will be automatically detected by MoviePy, except on Windows! Before installing MoviePy by hand, Windows users need to edit moviepy/config_defaults.py to provide the path to the ImageMagick binary, which is called convert. It should look like this: IMAGEMAGICK_BINARY = "C:\\Program Files\\ImageMagick_VERSION\\convert.exe".

Nathan Tuggy
  • 2,237
  • 27
  • 30
  • 38
zpengl
  • 61
  • 1
  • 2