17

I have been using selenium chromedriver in google colab for a while.
Today I start my work, I get the error never got before.
I'm wondering if this error is related to the chromedriver version.

This is install log today

!apt-get update
!apt install -y chromium-chromedriver


Get:1  focal-updates/main amd64 apparmor amd64 2.13.3-7ubuntu5.1 [494 kB]
Get:2  focal/main amd64 liblzo2-2 amd64 2.10-2 [50.8 kB]
Get:3  focal-updates/main amd64 squashfs-tools amd64 1:4.4-1ubuntu0.3 [117 kB]
Get:4  focal-updates/main amd64 snapd amd64 2.57.5+20.04ubuntu0.1 [37.6 MB]
Get:5  focal-updates/universe amd64 chromium-browser amd64 1:85.0.4183.83-0ubuntu0.20.04.2 [48.3 kB]
Get:6  focal-updates/universe amd64 chromium-chromedriver amd64 1:85.0.4183.83-0ubuntu0.20.04.2 
...
...
(Reading database ... 129791 files and directories currently installed.)
Preparing to unpack .../chromium-browser_1%3a85.0.4183.83-0ubuntu0.20.04.2_amd64.deb ...
=> Installing the chromium snap
==> Checking connectivity with the snap store
===> System doesn't have a working snapd, skipping
Unpacking chromium-browser (1:85.0.4183.83-0ubuntu0.20.04.2) ...
Selecting previously unselected package chromium-chromedriver.
Preparing to unpack .../chromium-chromedriver_1%3a85.0.4183.83-0ubuntu0.20.04.2_amd64.deb ...
Unpacking chromium-chromedriver (1:85.0.4183.83-0ubuntu0.20.04.2) ...
Setting up chromium-browser (1:85.0.4183.83-0ubuntu0.20.04.2) ...
update-alternatives: using /usr/bin/chromium-browser to provide /usr/bin/x-www-browser (x-www-browser) in auto mode
update-alternatives: using /usr/bin/chromium-browser to provide /usr/bin/gnome-www-browser (gnome-www-browser) in auto mode
Setting up chromium-chromedriver (1:85.0.4183.83-0ubuntu0.20.04.2) ...
Processing triggers for mime-support (3.64ubuntu1) ...
Processing triggers for hicolor-icon-theme (0.17-2) ...
Processing triggers for libc-bin (2.31-0ubuntu9.9) ...
Processing triggers for systemd (245.4-4ubuntu3.19) ...
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for dbus (1.12.16-2ubuntu2.3) ...

This is install log yesterday

Get:1  bionic-updates/universe amd64 chromium-codecs-ffmpeg-extra amd64 108.0.5359.71-0ubuntu0.18.04.5 [1,159 kB]
Get:2  bionic-updates/universe amd64 chromium-browser amd64 108.0.5359.71-0ubuntu0.18.04.5 [83.6 MB]
Get:3  bionic-updates/universe amd64 chromium-browser-l10n all 108.0.5359.71-0ubuntu0.18.04.5 [5,230 kB]
Get:4  bionic-updates/universe amd64 chromium-chromedriver amd64 108.0.5359.71-0ubuntu0.18.04.5 [5,594 kB]
...
...
Setting up chromium-chromedriver (108.0.5359.71-0ubuntu0.18.04.5) ...
Setting up chromium-browser-l10n (108.0.5359.71-0ubuntu0.18.04.5) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for hicolor-icon-theme (0.17-2) ...
Processing triggers for mime-support (3.60ubuntu1) ...
Processing triggers for libc-bin (2.27-3ubuntu1.6) ...

I found the colab system is updated to ubuntu 20.04 from 18.04.
Is this the reason why chromedriver cannot be used?
How should I do to solve this problem?
Thanks.

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
Wayne Gao
  • 173
  • 1
  • 5

4 Answers4

9

These log messages...

Get:1  bionic-updates/universe amd64 chromium-codecs-ffmpeg-extra amd64 108.0.5359.71-0ubuntu0.18.04.5 [1,159 kB]
Get:2  bionic-updates/universe amd64 chromium-browser amd64 108.0.5359.71-0ubuntu0.18.04.5 [83.6 MB]
Get:3  bionic-updates/universe amd64 chromium-browser-l10n all 108.0.5359.71-0ubuntu0.18.04.5 [5,230 kB]
Get:4  bionic-updates/universe amd64 chromium-chromedriver amd64 108.0.5359.71-0ubuntu0.18.04.5 [5,594 kB]
...
...
Setting up chromium-chromedriver (108.0.5359.71-0ubuntu0.18.04.5) ...
Setting up chromium-browser-l10n (108.0.5359.71-0ubuntu0.18.04.5) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for hicolor-icon-theme (0.17-2) ...
Processing triggers for mime-support (3.60ubuntu1) ...
Processing triggers for libc-bin (2.27-3ubuntu1.6) ...

...is the result of an issue induced as the colab system is updated from v18.04 to ubuntu 20.04 LTS recently.

Quick Fix

@mco-gh created a new notebook following @metrizable's guidance (details below) which is working perfect as of now: https://colab.research.google.com/drive/1cbEvuZOhkouYLda3RqiwtbM-o9hxGLyC

Details

In the discussion Issues when trying to use Chromedriver in Colab @EvanWiederspan clearly mentioned:

We recently upgraded the Ubuntu version, which may be causing this. After connecting to a runtime, can you try running "Use fallback runtime version" from the command palette (ctrl + shift + p)

@metrizable in his comment further clarified:

I understand that you'd like to use selenium with chromium-browser under Ubuntu 20.04 LTS. Although YMMV, since Ubuntu 20.04+ no longer distributes chromium-browser outside of a snap package, you can install a compatible version from the Debian buster repository

Sample code:

%%shell
# Ubuntu no longer distributes chromium-browser outside of snap
#
# Proposed solution: https://askubuntu.com/questions/1204571/how-to-install-chromium-without-snap

# Add debian buster
cat > /etc/apt/sources.list.d/debian.list <<'EOF'
deb [arch=amd64 signed-by=/usr/share/keyrings/debian-buster.gpg] http://deb.debian.org/debian buster main
deb [arch=amd64 signed-by=/usr/share/keyrings/debian-buster-updates.gpg] http://deb.debian.org/debian buster-updates main
deb [arch=amd64 signed-by=/usr/share/keyrings/debian-security-buster.gpg] http://deb.debian.org/debian-security buster/updates main
EOF

# Add keys
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys DCC9EFBF77E11517
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 648ACFD622F3D138
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 112695A0E562B32A

apt-key export 77E11517 | gpg --dearmour -o /usr/share/keyrings/debian-buster.gpg
apt-key export 22F3D138 | gpg --dearmour -o /usr/share/keyrings/debian-buster-updates.gpg
apt-key export E562B32A | gpg --dearmour -o /usr/share/keyrings/debian-security-buster.gpg

# Prefer debian repo for chromium* packages only
# Note the double-blank lines between entries
cat > /etc/apt/preferences.d/chromium.pref << 'EOF'
Package: *
Pin: release a=eoan
Pin-Priority: 500


Package: *
Pin: origin "deb.debian.org"
Pin-Priority: 300


Package: chromium*
Pin: origin "deb.debian.org"
Pin-Priority: 700
EOF

# Install chromium and chromium-driver
apt-get update
apt-get install chromium chromium-driver

# Install selenium
pip install selenium

Note

Generally the following line of code will install the snap too:

sudo apt install chromium-chromedriver

incase it doesn't you have to manually install the snap using:

sudo snap install chromium
undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
  • You might format the sample code as a shell-codebox? – kaliiiiiiiii Jan 19 '23 at 06:48
  • Also Note, that snap doesn't seem to work using colab. – kaliiiiiiiii Jan 19 '23 at 07:09
  • 3
    @kaliiiiiiiii I just transfer the debian.list and chromium.pref as file to google driver. when notebook start I copy these files to current path from my google driver. Then add keys and install chromedriver. It's works for me. – Wayne Gao Jan 19 '23 at 07:43
  • This answer deserves more upvotes imho (it was plagiated several times since being posted) -- this helped a lot of people using Colab && Selenium at work to keep doing their jobs. +1. – Barry the Platipus Jan 19 '23 at 19:01
  • 1
    This did not work for me. I keep getting the same error. Message: Service chromedriver unexpectedly exited. Status code was: 1 Can someone, please help me? – user3347814 Jan 20 '23 at 14:23
2

You can use this sample colab workbook to rebuild your script. The reason is that the last Ubuntu update update supports chromium diver just via snap.

Post with explanation: https://github.com/googlecolab/colabtools/issues/3347#issuecomment-1397277515

Direct Link to Notebook: https://colab.research.google.com/drive/1cbEvuZOhkouYLda3RqiwtbM-o9hxGLyC

Bug Alpha
  • 13
  • 1
  • 5
Johannes Stephan
  • 55
  • 11
  • 1
    (Using Google Colab) It worked for me aswell but only after adding a line of code to the first cell in your notebook to remove chromium from Ubuntu first. (sudo apt remove chromium-browser chromium-browser-l10n chromium-codecs-ffmpeg-extra). See https://askubuntu.com/questions/1204571/how-to-install-chromium-without-snap – Torakoro Jan 21 '23 at 00:53
1

After the runtime status says connected, open the command palette (ctrl + shift + p) and select "Use Fallback Runtime Version"

That should resolve this issue!

Reference: https://github.com/googlecolab/colabtools/issues/3347#issuecomment-1386349011

We recently upgraded the Ubuntu version, which may be causing this. After connecting to a runtime, can you try running "Use fallback runtime version" from the command palette (ctrl + shift + p) and seeing if your notebook works after that?

rfausak
  • 11
  • 2
1

Better solution is to do this first in addition to your usual thing you do in colab to make selenium work:

!sudo add-apt-repository ppa:saiarcot895/chromium-beta

!sudo apt remove chromium-browser
!sudo snap remove chromium

!sudo apt install chromium-browser

Note: Solution provided here by @Johannes Stephan works but it crashes browser a lot and makes it unusable

FULL Code:

!sudo add-apt-repository ppa:saiarcot895/chromium-beta
!sudo apt remove chromium-browser
!sudo snap remove chromium
!sudo apt install chromium-browser

!pip3 install selenium
!apt-get update
!apt install chromium-chromedriver
!cp /usr/lib/chromium-browser/chromedriver /usr/bin

import sys
sys.path.insert(0,'/usr/lib/chromium-browser/chromedriver')
yf879
  • 168
  • 1
  • 7