Memory leaks in a python script, Xvfb is suspected

Question

On a linux server I'm using Chrome/Selenium with Xvfb in a python script. Sometimes the script crashes for other reason and thus, according to what I see in the dashboard of digital ocean, the ram consumption of "Xvfb" increases up to almost 80% eventually, over time. However, the ram increase isn't necessarily due to the script being crashed, but it might be because of an wrong of releasing Xvfb in general.

Here's what I have relevant to xvfb

    from pyvirtualdisplay import Display
    ..........
    display = Display(visible=0, size=(800, 600))
    display.start()


    //it can crash here doing other things

    display.sendstop()

Am I releasing Xvfb properly in the first place?
When it crashes, does Xvfb get released automatically or not?
Should I wrap my code into try ... except to be able properly release Xvfb?

joelnb · Accepted Answer · 2016-12-21T00:30:24.370

Short answer is that display.sendstop() is not being called if your code crashes where you showed.

I think the most 'pythonic' way to achieve what you want would be to use the Display as a context manager, meaning that you do not need to wrap all of your code in a try/catch but you get the same benefit. Something like this should work:

import pyvirtualdisplay

with pyvirtualdisplay.Display(visible=0, size=(800, 600)):
    // it can crash here doing other things

EDIT: The reason that the call to the stop method is important in this particular case (and why you are almost certainly right about the memory leak) is because your code is spawning an Xvfb subprocess to act as the virtual display. The stop method kills this subprocess and so if it is never called the subprocess remains running & is reparented onto the init process. There is some more information on this here (section 'Zombie and Orphan Processes').

I was able to check how this works using this simple bit of code:

import time
from pyvirtualdisplay import Display

display = Display(visible=0, size=(800, 600))
display.start()

print('About to sleep')
time.sleep(20)

raise Exception('Oh noes!')

display.stop()

I then ran pstree -sA $(pgrep Xvfb) in my shell (to show the process tree of the Xvfb process) before and after the python script crashed.

Before the crash we can see that the parent process of the Xvfb process is python (whose parent is my shell, whose parent is my terminal emulator etc.):

systemd---xfce4-terminal---zsh---python---Xvfb-+-{llvmpipe-0}
                                               |-{llvmpipe-1}
                                               |-{llvmpipe-2}
                                               `-{llvmpipe-3}

After the python script crashes the process has now been reparented onto the init process (in my case this is systemd but in your case it may be some other init system).

systemd---Xvfb-+-{llvmpipe-0}
               |-{llvmpipe-1}
               |-{llvmpipe-2}
               `-{llvmpipe-3}

`Short answer is that display.sendstop() is not being called if your code crashes where you showed.` -- suppose it's not. but since the script has crashed, don't the OS close all the handlers it's opened? — Saurabh, Dec 20 '16 at 09:45
`Short answer is that display.sendstop() is not being called if your code crashes where you showed.` -- suppose it's not. but since the script has crashed, doesn't the OS close all the handlers it's opened? — Saurabh, Dec 20 '16 at 09:46
I hope the edit I have made explains this better. In short the answer is no - PyVirtualDisplay is creating a subprocess which remains even after the script has exited. — joelnb, Dec 21 '16 at 00:33
so the correct way to release the xfvd resource is to call "stop()", right? — Saurabh, Dec 21 '16 at 05:24
Yes, or use the `Display` as a context manager like I showed in my answer. As long as you make sure `stop()` is ALWAYS called you will be okay. — joelnb, Dec 21 '16 at 08:44

Memory leaks in a python script, Xvfb is suspected

1 Answers1