1

I am using Selenium to open a certain website (for example YouTube) on the server, but it can't seem to open the website. However, the code works just fine with a different website. This code also works fine without any problems on my local PC.

I don't know if I have problems with my Chrome Driver or Selenium but it can't open youtube.com as it only outputs: "Before getting the website" and that's it. There are no exceptions/errors that are shown but the script still runs and I have to manually end stop it.

Why can't Selenium open certain URLs on the server, but it works fine on my PC?

options = webdriver.ChromeOptions()
options.add_argument("no-sandbox")
options.add_argument('--headless')
options.add_argument("--start-maximized")
PATH = "./chromedriver"
global driver
driver = webdriver.Chrome(PATH, chrome_options=options)

print("Before getting the website")
driver.get("https://youtube.com")
print("opened", driver.current_url)
mkrieger1
  • 19,194
  • 5
  • 54
  • 65
LilReef599
  • 31
  • 4
  • Just a wild guess: Is the cloud environment by Amazon maybe deliberately blocking those requests? Maybe they have some sort of anti-scraping policy? – Marcel Aug 12 '20 at 19:40
  • I suggest enabling verbose logging for webdriver as per [this article](https://chromedriver.chromium.org/logging). Maybe something helpful would show up there. – Marek Piotrowski Aug 12 '20 at 20:12

1 Answers1

0

I had a exactly same problem. Perhaps i don't know why this happened.

NOTE:

When you are scraping working on Ubuntu ec2 with no GUI you have to provide some GUI interface for chrome to run and for me Xvfb solved it. "Xvfb (short for X virtual framebuffer) is an in-memory display server for UNIX-like operating system (e.g., Linux). It enables you to run graphical applications without a display (e.g., browser tests on a CI server) while also having the ability to take screenshots."

SOLUTION

  1. Install Xvfb for ubuntu: sudo apt install xvfb
  2. Now execute your script as: xvfb-run python[version] script.py

IMPORTANT NOTE:

If your program stuck during initialization does not show any output just make sure that you did not add chrome_option.add_argument("disable-dev-shm-usage"). If this argument is added in your chrome header then it would disable the /dev/shm. Not sure but it is some shared memory and xvfb is in-memory display server which i think need it. This worked for me.

Alexo
  • 16
  • 2