2

Overview:

Selenium scraper works perfectly in headless mode. Spawning a virtual display shows no errors via XVFB:

from xvfbwrapper import Xvfb

vdisplay = Xvfb()
vdisplay.start()
vdisplay.stop()

But when I try to run them together, it errors out with:

[ERROR] SessionNotCreatedException: Message: session not created from disconnected: Unable to receive message from renderer (Session info: chrome=96.0.4664.0)

Traceback:

Traceback (most recent call last):
  File "/var/task/slack_main.py", line 34, in handler
    scrape_price(asin_list)
  File "/var/task/slack_main.py", line 58, in scrape_price
    driver = webdriver.Chrome("/opt/chromedriver",options=options)
  File "/var/lang/lib/python3.9/site-packages/selenium/webdriver/chrome/webdriver.py", line 70, in __init__
    super(WebDriver, self).__init__(DesiredCapabilities.CHROME['browserName'], "goog",
  File "/var/lang/lib/python3.9/site-packages/selenium/webdriver/chromium/webdriver.py", line 92, in __init__
    RemoteWebDriver.__init__(
  File "/var/lang/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 275, in __init__
    self.start_session(capabilities, browser_profile)
  File "/var/lang/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 365, in start_session
    response = self.execute(Command.NEW_SESSION, parameters)
  File "/var/lang/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 430, in execute
    self.error_handler.check_response(response)
  File "/var/lang/lib/python3.9/site-packages/selenium/webdriver/remote/errorhandler.py", line 247, in check_response
    raise exception_class(message, screen, stacktrace)

Configuration:

Below is my complete Selenium and XVFB configuration:

from selenium import webdriver
from selenium_stealth import stealth
from xvfbwrapper import Xvfb

vdisplay = Xvfb()
vdisplay.start()
options = webdriver.ChromeOptions()
prefs = {"browser.downloads.dir": "//tmp//", "download.default_directory": "//tmp//", "directory_upgrade": True}
options.add_experimental_option("prefs", prefs)
options.binary_location = '/opt/chrome/chrome'
#options.add_argument('--headless') #toggled on and off when running with or without XVFB
options.add_argument('--no-sandbox')
options.add_argument("--disable-gpu")
options.add_argument("--window-size=1280x1696")
options.add_argument("--single-process")
options.add_argument("--disable-dev-shm-usage")
options.add_argument("--disable-dev-tools")
options.add_argument("--no-zygote")
options.set_capability('unhandledPromptBehavior', 'ignore')
options.add_argument("download.default_directory=/tmp")
driver = webdriver.Chrome("/opt/chromedriver",options=options)
stealth(driver,
        languages=["en-US", "en"],
        vendor="Google Inc.",
        platform="Win32",
        fix_hairline=True,
        )
vdisplay.stop()
driver.close()

Why is it not connecting to the display? My guess is it has something to do with the '--headless' toggle?

Versions and Tools:

  • Selenium version 3.141.0
  • xvfbwrapper version 0.2.9
  • Docker is used to compile and push to AWS Lambda, base image used (no changes are made in docker file with or without XVFB)

Edit:

Found a pull request for XVFB configuration in the Github Repo of my base image. Even used the exact same code from the pull request and I still recieve the exact same error. Maybe this has something to do with AWS offboard?

Luke Hamilton
  • 637
  • 5
  • 19
  • Could you share your traceback and the exact error message? – dosas Jun 21 '22 at 11:20
  • @PhilippSelenium Thank you for that suggestion, I have updated my question with the traceback and all the non-encoding error messages – Luke Hamilton Jun 21 '22 at 13:27
  • Are you sure that you are using the correct version of the chromedriver 96.0.4664.0. You should check chromedriver --version and google-chrome --version match. – dosas Jun 22 '22 at 07:34
  • BTW: The code you posted works perfectly fine for me locally – dosas Jun 22 '22 at 07:35
  • @dosas wouldn't ChromeDriver not launch in the first place if this is a version error? Selenium scraper works perfectly fine without the XVFB display running. Seems to be more of a display connection issue to me. Also note: I added more details in the "edit" section. – Luke Hamilton Jun 22 '22 at 22:27

0 Answers0