1

I have got github codespaces environment and I have installed both selenium and the necessary chromedriver-binary using pip

pip install selenium chromedriver-binary

Here's an example of the Python web scraper I'm writing

import json
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service

class PriceScraper:

    def scrape(self):

        input_url = "https://www.google.com"
        chrome_options = Options()
        chrome_options.add_argument("--headless")
        chrome_options.add_argument("--disable-dev-shm-usage")
        chrome_options.add_argument("--no-sandbox")
        service = Service('/usr/bin/chromedriver')
        driver = webdriver.Chrome(service=service, options=chrome_options)
        driver.get(input_url)

if __name__ == '__main__':
    scraper = PriceScraper()
    scraper.scrape()

I have installed all the necessary pip packages and I have confirmed the installation of the chromium and the chromedriver by running:

(venv) $   sudo apt-get install -y chromium-browser chromium-chromedriver python3-selenium
Reading package lists... Done
Building dependency tree       
Reading state information... Done
python3-selenium is already the newest version (4.0.0~a1+dfsg1-1.1).
chromium-browser is already the newest version (1:85.0.4183.83-0ubuntu0.20.04.3).
chromium-chromedriver is already the newest version (1:85.0.4183.83-0ubuntu0.20.04.3).

And checking by running ls -l /usr/bin/chromedriver

But when I try to execute the python from my vscode codespaces terminal as follows:

python3 scrape_prices.py

It returns the following error:

selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: exited abnormally.
  (unknown error: DevToolsActivePort file doesn't exist)
  (The process started from chrome location /usr/bin/chromium-browser is no longer running, so ChromeDriver is assuming that Chrome has crashed.)

Interestingly when I try to run chromedriver from the command line it says:

Command '/usr/bin/chromedriver' requires the chromium snap to be installed.
Please install it with:

snap install chromium

And when I attempt to install snap with snap install chromium

I get the following error

error: cannot communicate with the server: Post http://localhost/v2/snaps/chromium: dial unix /run/snapd.socket: connect: no such file or directory

I'm unsure how to get this working

steambun
  • 35
  • 7
  • 1
    you haven't installed chrome yet, in order to use selenium you need to install chrome browser too – EvilReboot Apr 29 '23 at 07:07
  • I have amended the question to add that I had installed both chromium and the driver - see notes above. – steambun Apr 29 '23 at 09:35
  • the best way will be to use a portable version of chromium, download portable chromium attach it to the script and modify `options` parameter – EvilReboot Apr 29 '23 at 10:09
  • another way can be by using docker container. you can use containerised chrome or firefox – EvilReboot Apr 29 '23 at 18:33

2 Answers2

0

Seems like Snapd isn't installed properly, and because of that also Chromium

Got that issue some time ago with google-colab as well, the Solution was following:

Ubuntu 20.04+ no longer distributes chromium-browser outside of a snap package, you can install a compatible version from the Debian buster repository:

%%shell
# Ubuntu no longer distributes chromium-browser outside of snap
#
# Proposed solution: https://askubuntu.com/questions/1204571/how-to-install-chromium-without-snap

# Add debian buster
cat > /etc/apt/sources.list.d/debian.list <<'EOF'
deb [arch=amd64 signed-by=/usr/share/keyrings/debian-buster.gpg] http://deb.debian.org/debian buster main
deb [arch=amd64 signed-by=/usr/share/keyrings/debian-buster-updates.gpg] http://deb.debian.org/debian buster-updates main
deb [arch=amd64 signed-by=/usr/share/keyrings/debian-security-buster.gpg] http://deb.debian.org/debian-security buster/updates main
EOF

# Add keys
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys DCC9EFBF77E11517
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 648ACFD622F3D138
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 112695A0E562B32A

apt-key export 77E11517 | gpg --dearmour -o /usr/share/keyrings/debian-buster.gpg
apt-key export 22F3D138 | gpg --dearmour -o /usr/share/keyrings/debian-buster-updates.gpg
apt-key export E562B32A | gpg --dearmour -o /usr/share/keyrings/debian-security-buster.gpg

# Prefer debian repo for chromium* packages only
# Note the double-blank lines between entries
cat > /etc/apt/preferences.d/chromium.pref << 'EOF'
Package: *
Pin: release a=eoan
Pin-Priority: 500


Package: *
Pin: origin "deb.debian.org"
Pin-Priority: 300


Package: chromium*
Pin: origin "deb.debian.org"
Pin-Priority: 700
EOF

# Install chromium and chromium-driver
apt-get update
apt-get install chromium chromium-driver

# Install selenium
pip install selenium

Example script:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument("--headless") # or use pyvirtualdiplay
options.add_argument("--no-sandbox") # needed, because colab runs as root

options.headless = True

driver = webdriver.Chrome("/usr/bin/chromedriver", options=options)

driver.get("https://github.com/kaliiiiiiiiii/Selenium-Profiles")
print(driver.title)
driver.quit()

resource: googlecolab/colabtools#3347 (comment)

See Google-Colab (file: google-colab/selenium_profiles.ipynb) for an example

resource

Disclosure: I am the author of Selenium-Profiles

kaliiiiiiiii
  • 925
  • 1
  • 2
  • 21
0

The issue was that I hadn't installed chromium locally, and once that was done properly from within my .venv virtual environment, everything worked properly.

steambun
  • 35
  • 7
  • 1
    Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community May 18 '23 at 06:24