12

I have a long running python app that will periodically (every 30-60 seconds) open a webpage with selenium and chrome driver, run some javascript and take a screenshot. Its running on an EC2 ubuntu instance with chrome in Xvfb and for the most part everything is working, except intermittently the program will hang. It is happening on one of these lines:

    options = Options()
    options.add_argument("--disable-web-security")
    options.add_argument("--webdriver-logfile=webdrive.log")
    dc = DesiredCapabilities.CHROME
    dc['loggingPrefs'] = {'browser': 'ALL'}
    driver = webdriver.Chrome(chrome_options=options, desired_capabilities=dc)
    driver.get(url);

(I don't have an exact line but I know from debug statements I have put in that it is somewhere in between here)

Unfortunately, the program doesn't crash so it doesn't have any error messages, its just waiting endlessly since 7pm last night. Running strace -p 'python program pid' returns: wait4(-1, and running strace -p 'chromedriver pid' returns recvfrom(20,

I can see in ps axjf that the process is still running, its just not doing anything. I'm sort of at a loss of what to do now, any suggestions?

chromedriver version: 2.10.267518

Google Chrome 40.0.2214.111

Selenium (installed with pip): 2.42.1

#https://github.com/cgoldberg/xvfbwrapper
xvfb = Xvfb(width=1920, height=1920)
xvfb.start()

---- EDIT ----

I have just updated to ChromeDriver 2.14.313457 and Selenium 2.44.0, hopefully this will fix the issue. I'm going to leave this open for now. Thanks for the advice so far guys!

---- EDIT ----

So the service still ended up hanging. I'm wondering if this is because for every screenshot I close and restart google-chrome? Is this possibly causing a memory leak somehow? How could I diagnose this?

Trevor
  • 995
  • 3
  • 10
  • 25
  • 1
    Does it hang if chrome options are not specified? Which selenium and chrome versions are you using? – alecxe Feb 11 '15 at 20:36
  • Also, try to add debug log messages and see on which line is it hanging. Also, show how do you start `xvfb`. Thanks. – alecxe Feb 11 '15 at 20:43
  • see my edits. I've been adding debug messages over the few times that this has happened, though I guess I wasn't thorough enough. – Trevor Feb 11 '15 at 20:47
  • Well, the first thing to try is to upgrade selenium to the latest (currently 2.44) version. – alecxe Feb 11 '15 at 20:50
  • I think the Chromedriver is the culprit here. Chromedriver 2.10 only covers up to Chrome version 36. – Richard Feb 11 '15 at 20:52
  • do you ever stop xvfb? or just keep spawning new ones? – Corey Goldberg Apr 21 '15 at 19:11

3 Answers3

11

I ran into a similar issue and found the answer here and blogged about it here. Setting the environment variable DBUS_SESSION_BUS_ADDRESS=/dev/null worked for me without having to restart Xvfb all the time.

Kevin Schroeder
  • 1,296
  • 11
  • 23
1

I never found out the specific piece of code that was causing this problem, but creating a fresh instance of Xvfb with each driver load seems to have fixed it. Perhaps there is a memory leak somewhere in the interaction between selenium and Xvfb? Either way, marking this as closed.

Trevor
  • 995
  • 3
  • 10
  • 25
0

In my case, the problem disappeared enclosing the call within a with block, like this:

options = Options()
path = f"{os.environ['LOCALAPPDATA']}{os.path.sep}Programs{os.path.sep}Chromium{os.path.sep}Win_x64{os.path.sep}1012736{os.path.sep}"
options.binary_location = f"{path}chrome-win{os.path.sep}chrome.exe"
executable_path = f"{path}chromedriver_win32{os.path.sep}chromedriver.exe"
with webdriver.Chrome(chrome_options=options, executable_path=executable_path) as browser:
    browser.get("https://live.euronext.com/en/products/equities")
    sleep(3)

Selenium 4.4.0, Python 3.10, Windows 10, ChromeDriver and Chromium 104.0.5112.0

Antonio Miras
  • 116
  • 6