15

I’m writing a web app that uses Selenium to screen-scrape another website. This screen-scraping only happens once a day, so I’d rather not leave Selenium and Xvfb running all the time.

I’m trying to figure out how to start Xvfb and Selenium from Python, and then stop them once the screen-scraping’s done.

If I was doing it manually, I’d start them at the command line, and hit CTRL C to stop them. I’m trying to do the same thing from Python.

I seem to be able to successfully start Xvfb like this:

xvfb = Popen('Xvfb :99 -nolisten tcp', shell=True)

But when I’ve tried to terminate it:

xvfb.terminate()

and then tried to start it again (by repeating my initial command), it tells me it’s already running.

Paul D. Waite
  • 96,640
  • 56
  • 199
  • 270
  • Try `xvfb.wait()` after `xvfb.terminate()`. Failing that, try `xvfb.kill()`. – Thomas K Apr 01 '11 at 17:14
  • 3
    Ctrl-C sends SIGTERM to a Unix process. Your user python process cannot send SIGTERM to your Xvfb running as root. Get your child's pid, execute "sudo kill ", then "xvfb.wait()". – Spike Gronim Apr 01 '11 at 17:15
  • @Spike: ah, okay, that makes sense. Maybe I should at least have Xvfb running all the time, as it seems to need a sudoer to start it, and that, I imagine, is going to be a bit awkward from Python? (So far I’ve just been typing my password in at the Python shell, which obviously won’t work when this is normally running.) – Paul D. Waite Apr 01 '11 at 17:21

2 Answers2

7

I don't know why you want to run Xvfb as root. Your usual X server only needs to run as root (on many but not all unices) only so that it can access the video hardware; that's not an issue for Xvfb by definition.

tempdir = tempfile.mkdtemp()
xvfb = subprocess.Popen(['Xvfb', ':99', '-nolisten', 'tcp', '-fbdir', tempdir])

When you terminate the X server, you may see a zombie process. This is in fact not a process (it's dead), just an entry in the process table that goes away when the parent process either reads the child's exit status or itself dies. Zombies are mostly harmless, but it's cleaner to call wait to read the exit status.

xvfb.terminate()
# At this point, `ps -C Xvfb` may still show a running process
# (because signal delivery is asynchronous) or a zombie.
xvfb.wait()
# Now the child is dead and reaped (assuming it didn't catch SIGTERM).
Gilles 'SO- stop being evil'
  • 104,111
  • 38
  • 209
  • 254
  • @Gilles: ah, gotcha — I’d copied and pasted the command I use to start Xvfb from a web page somewhere. I didn’t realise it was the `fbdir` option causing it to not work if I didn’t run it as root. I’ve removed that now and can run it as my usual user, but `.terminate()` and `.kill()` still don’t seem to stop Xvfb. – Paul D. Waite Apr 01 '11 at 21:27
  • @Paul: I made a couple of typos in the `Popen` call but I think you got it right. Are you sure `.terminate()` and `.kill()` aren't terminating the process? A zombie may remain (it would disappear when your Python script terminates), see my edit. – Gilles 'SO- stop being evil' Apr 01 '11 at 21:46
  • @Gilles: Yeah, it looks like the process is hanging around. After `.terminate()`, `.kill()`, and after I exit the Python shell, `ps -C Xvfb` still reports that Xvfb is running. When I attempt to start Xvfb again, it also says the server is already active. (I think that’s just because the file `/tmp/.X99-lock` still exists, but `ps -C Xvfb` confirms that the process is still running.) – Paul D. Waite Apr 01 '11 at 22:41
  • @Gilles: **AHA** — but when I use `Popen` the way you suggested (i.e. with arguments, rather than a string and `shell=True`), then everything works just as you said. Phew! Many, many thanks for looking into this, I’m missing a fair bit of basic Unix/Linux knowledge, and so tend to get a bit stumped when trying to debug problems. May a thousand upvotes be granted to your answers. – Paul D. Waite Apr 01 '11 at 22:44
  • 1
    @Paul: Ah, so you were killing the shell but not Xvfb. If you needed shell expansion before calling Xvfb but not after, you could run `Popen("exec Xvfb …", shell=True)` so that the Xvfb process would replace the shell instead of running as a child process. If you needed the shell afterwards, you'd need to communicate the process ID of Xvfb to the python code somehow. But here, it's easiest all around to not use a shell. – Gilles 'SO- stop being evil' Apr 01 '11 at 22:48
  • @Gilles: *right* — that makes sense, I see where it was going wrong now. Definitely no need for a shell here. – Paul D. Waite Apr 01 '11 at 22:54
0

I assume you can parametrize your system to allow any user to launch Xvfb as explained here solving all your problems

EDIT the correct command line is 

sudo chmod u+s `which Xvfb`
Xavier Combelle
  • 10,968
  • 5
  • 28
  • 52
  • Could be — which part of that allows any user to launch Xvfb though? Is it `sudo mod u+s `which Xvfb``? I’m not familiar with the `mod` command, and it’s somewhat immune to Googling — could you expand on it? – Paul D. Waite Apr 01 '11 at 18:12
  • in fact it should be `chmod u+s` as explained here http://en.wikipedia.org/wiki/Chmod – Xavier Combelle Apr 01 '11 at 18:27
  • I doubt that the `mod` command exist to have help on a command just write `man command` – Xavier Combelle Apr 01 '11 at 18:43
  • @Xavier: ah! Indeed, that works perfectly for making Xvfb runnable by users who aren’t root. Unfortunately, `.terminate()` and `.kill()` still don’t seem to stop the process. – Paul D. Waite Apr 01 '11 at 18:47
  • @Paul you are right I was missing that in fact it makes a root program launch and identify as root. But your program still runs as root – Xavier Combelle Apr 01 '11 at 18:52
  • @Paul you can make your own program run as root even if it's quite dangerous – Xavier Combelle Apr 01 '11 at 18:54