I am using Pythons Selenium and Undetected Chrome for web-scraping. I have something like the following code:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
import random
from bs4 import BeautifulSoup
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.chrome.options import Options
import undetected_chromedriver as uc
import sys
RUN_HEADLESS = True
PATH = '/home/matt/bin/chromedriver/chromedriver'
options = uc.ChromeOptions()
if RUN_HEADLESS:
options.add_argument('--headless')
else:
pass
driver = uc.Chrome(options=options, executable_path=PATH)
driver.delete_all_cookies()
driver.get("https://www.google.com/")
driver.maximize_window()
... # Code to scrape website
driver.close()
driver.quit()
Everything works well and I have a cron job to run the code every 10 minutes. However, in the directory /home/myUSER/.local/share/undetected_chromedriver/ I have many instances of undetected chrome.
This has a few side-affects. It slows down the computer when its been running for a few hours and the undetected chrome driver code breaks.
One error is:
[0510/015403.947127:ERROR:file_io_posix.cc(152)] open /home/matt/.config/google-chrome/Crash Reports/pending/390c0413-938f-4885-a704-658eeb19a41b.lock: File exists (17) [30118:30196:0510/015404.009286:ERROR:file_path_watcher_inotify.cc(315)] inotify_init() failed: Demasiados ficheros abiertos (24)
DevTools listening on ws://127.0.0.1:35347/devtools/browser/5fb5c0ae-5b38-455b-954f-a6292102d669
MESA-INTEL: warning: Performance support disabled, consider sysctl dev.i915.perf_stream_paranoid=0
[30222:30481:0510/015404.088775:ERROR:file_path_watcher_inotify.cc(315)] inotify_init() failed: Demasiados ficheros abiertos (24)
[0510/015404.774902:ERROR:file_io_posix.cc(152)] open /home/matt/.config/google-chrome/Crash Reports/pending/390c0413-938f-4885-a704-658eeb19a41b.lock: File exists (17)
[0510/015404.871718:ERROR:file_io_posix.cc(152)] open /home/matt/.config/google-chrome/Crash Reports/pending/390c0413-938f-4885-a704-658eeb19a41b.lock: File exists (17)
[30221:30221:0510/015416.083125:ERROR:gl_utils.cc(402)] [.WebGL-0x1fac0423f100]GL Driver Message (OpenGL, Performance, GL_CLOSE_PATH_NV, High): GPU stall due to ReadPixels
[31310:31310:0510/015419.264976:ERROR:platform_thread_posix.cc(155)] pthread_create: Recurso no disponible temporalmente (11)
[31298:31298:0510/015419.269948:ERROR:platform_thread_posix.cc(155)] pthread_create: Recurso no disponible temporalmente (11)
[31330:31330:0510/015419.271520:ERROR:platform_thread_posix.cc(155)] pthread_create: Recurso no disponible temporalmente (11)
[31327:31327:0510/015419.271845:ERROR:platform_thread_posix.cc(155)] pthread_create: Recurso no disponible temporalmente (11)
[31310:31310:0510/015419.275153:ERROR:platform_thread_posix.cc(155)] pthread_create: Recurso no disponible temporalmente (11)
[0510/015419.301093:ERROR:file_io_posix.cc(152)] open /home/matt/.config/google-chrome/Crash Reports/pending/390c0413-938f-4885-a704-658eeb19a41b.lock: File exists (17)
[0510/015419.313842:ERROR:file_io_posix.cc(152)] open /home/matt/.config/google-chrome/Crash Reports/pending/390c0413-938f-4885-a704-658eeb19a41b.lock: File exists (17)
[0510/015419.327857:ERROR:file_io_posix.cc(152)] open /home/matt/.config/google-chrome/Crash Reports/pending/390c0413-938f-4885-a704-658eeb19a41b.lock: File exists (17)
[0510/015419.342299:ERROR:file_io_posix.cc(152)] open /home/matt/.config/google-chrome/Crash Reports/pending/390c0413-938f-4885-a704-658eeb19a41b.lock: File exists (17)
[31016:31016:0510/015445.059161:ERROR:platform_thread_posix.cc(155)] pthread_create: Recurso no disponible temporalmente (11)
[31016:31025:0510/015445.059132:ERROR:platform_thread_posix.cc(155)] pthread_create: Recurso no disponible temporalmente (11)
[31016:31028:0510/015445.059133:ERROR:platform_thread_posix.cc(155)] pthread_create: Recurso no disponible temporalmente (11)
[31016:31025:0510/015445.059281:ERROR:platform_thread_posix.cc(155)] pthread_create: Recurso no disponible temporalmente (11)
[31016:31028:0510/015445.059286:ERROR:platform_thread_posix.cc(155)] pthread_create: Recurso no disponible temporalmente (11)
[31253:31290:0510/015449.595147:ERROR:platform_thread_posix.cc(155)] pthread_create: Recurso no disponible temporalmente (11)
[31253:31291:0510/015449.595147:ERROR:platform_thread_posix.cc(155)] pthread_create: Recurso no disponible temporalmente (11)
[31253:31288:0510/015449.595148:ERROR:platform_thread_posix.cc(155)] pthread_create: Recurso no disponible temporalmente (11)
[31253:31290:0510/015449.595385:ERROR:platform_thread_posix.cc(155)] pthread_create: Recurso no disponible temporalmente (11)
[30118:30217:0510/015505.060534:ERROR:zygote_communication_linux.cc(158)] Did not receive ping from zygote child
[30172:30172:0510/015505.060553:ERROR:zygote_linux.cc(622)] Zygote could not fork: process_type utility numfds 5 child_pid -1
[30118:30217:0510/015505.060551:ERROR:(-1)] Check failed: false.
[30172:30172:0510/015505.098127:ERROR:zygote_linux.cc(273)] Unexpected real PID message from browser
[0510/015505.098215:ERROR:file_io_posix.cc(152)] open /home/matt/.config/google-chrome/Crash Reports/pending/390c0413-938f-4885-a704-658eeb19a41b.lock: File exists (17)
[30172:30172:0510/015505.098204:ERROR:(-1)] Check failed: false.
[30118:30217:0510/015505.979611:ERROR:zygote_communication_linux.cc(158)] Did not receive ping from zygote child
[30118:30217:0510/015505.979632:ERROR:(-1)] Check failed: false.
[30172:30172:0510/015505.979613:ERROR:zygote_linux.cc(622)] Zygote could not fork: process_type utility numfds 5 child_pid -1
[30172:30172:0510/015505.979849:ERROR:zygote_linux.cc(273)] Unexpected real PID message from browser
[30172:30172:0510/015505.980295:ERROR:(-1)] Check failed: false.
[30118:30217:0510/015507.470514:ERROR:zygote_communication_linux.cc(158)] Did not receive ping from zygote child
[30118:30217:0510/015507.470534:ERROR:(-1)] Check failed: false.
[30172:30172:0510/015507.470517:ERROR:zygote_linux.cc(622)] Zygote could not fork: process_type utility numfds 5 child_pid -1
[30172:30172:0510/015507.470661:ERROR:zygote_linux.cc(273)] Unexpected real PID message from browser
[30172:30172:0510/015507.470737:ERROR:(-1)] Check failed: false.
[30118:30217:0510/015509.276041:ERROR:zygote_communication_linux.cc(158)] Did not receive ping from zygote child
[30118:30217:0510/015509.276061:ERROR:(-1)] Check failed: false.
[30172:30172:0510/015509.276043:ERROR:zygote_linux.cc(622)] Zygote could not fork: process_type utility numfds 5 child_pid -1
[30172:30172:0510/015509.276182:ERROR:zygote_linux.cc(273)] Unexpected real PID message from browser
[30172:30172:0510/015509.276238:ERROR:(-1)] Check failed: false.
[30118:30217:0510/015511.124547:ERROR:zygote_communication_linux.cc(158)] Did not receive ping from zygote child
[30118:30217:0510/015511.124566:ERROR:(-1)] Check failed: false.
[30172:30172:0510/015511.124549:ERROR:zygote_linux.cc(622)] Zygote could not fork: process_type utility numfds 5 child_pid -1
[30172:30172:0510/015511.124688:ERROR:zygote_linux.cc(273)] Unexpected real PID message from browser
[30172:30172:0510/015511.124771:ERROR:(-1)] Check failed: false.
[30118:30217:0510/015513.016716:ERROR:zygote_communication_linux.cc(158)] Did not receive ping from zygote child
[30118:30217:0510/015513.016736:ERROR:(-1)] Check failed: false.
[30172:30172:0510/015513.016718:ERROR:zygote_linux.cc(622)] Zygote could not fork: process_type utility numfds 5 child_pid -1
[30172:30172:0510/015513.016854:ERROR:zygote_linux.cc(273)] Unexpected real PID message from browser
[30172:30172:0510/015513.016939:ERROR:(-1)] Check failed: false.
[30118:30217:0510/015513.085972:ERROR:zygote_communication_linux.cc(158)] Did not receive ping from zygote child
[30118:30217:0510/015513.085989:ERROR:(-1)] Check failed: false.
[30172:30172:0510/015513.085976:ERROR:zygote_linux.cc(622)] Zygote could not fork: process_type utility numfds 5 child_pid -1
[30172:30172:0510/015513.086112:ERROR:zygote_linux.cc(273)] Unexpected real PID message from browser
[30172:30172:0510/015513.086175:ERROR:(-1)] Check failed: false.
[30118:30217:0510/015513.183591:ERROR:zygote_communication_linux.cc(158)] Did not receive ping from zygote child
[30118:30217:0510/015513.183607:ERROR:(-1)] Check failed: false.
[30172:30172:0510/015513.183594:ERROR:zygote_linux.cc(622)] Zygote could not fork: process_type utility numfds 5 child_pid -1
[30172:30172:0510/015513.183769:ERROR:zygote_linux.cc(273)] Unexpected real PID message from browser
[30172:30172:0510/015513.183861:ERROR:(-1)] Check failed: false.
[30118:30217:0510/015513.305466:ERROR:zygote_communication_linux.cc(158)] Did not receive ping from zygote child
[30118:30217:0510/015513.305483:ERROR:(-1)] Check failed: false.
[30172:30172:0510/015513.305470:ERROR:zygote_linux.cc(622)] Zygote could not fork: process_type utility numfds 5 child_pid -1
[30172:30172:0510/015513.305606:ERROR:zygote_linux.cc(273)] Unexpected real PID message from browser
[30172:30172:0510/015513.305693:ERROR:(-1)] Check failed: false.
[30118:30217:0510/015513.596025:ERROR:zygote_communication_linux.cc(158)] Did not receive ping from zygote child
[30118:30217:0510/015513.596043:ERROR:(-1)] Check failed: false.
[30172:30172:0510/015513.596028:ERROR:zygote_linux.cc(622)] Zygote could not fork: process_type utility numfds 5 child_pid -1
[30172:30172:0510/015513.596165:ERROR:zygote_linux.cc(273)] Unexpected real PID message from browser
[30172:30172:0510/015513.596338:ERROR:(-1)] Check failed: false.
[30580:30580:0510/015513.875240:ERROR:platform_thread_posix.cc(155)] pthread_create: Recurso no disponible temporalmente (11)
[0510/015513.914607:ERROR:file_io_posix.cc(152)] open /home/matt/.config/google-chrome/Crash Reports/pending/390c0413-938f-4885-a704-658eeb19a41b.lock: File exists (17)
Scrolling for: 1 seconds
Scrolling for: 2 seconds
Scrolling for: 3 seconds
Scrolling for: 4 seconds
Scrolling for: 5 seconds
Scrolling for: 6 seconds
Scrolling for: 7 seconds
Scrolling for: 8 seconds
Scrolling for: 9 seconds
Scrolling for: 10 seconds
Scrolling for: 11 seconds
Scrolling for: 12 seconds
Scrolling for: 13 seconds
Scrolling for: 14 seconds
Scrolling for: 15 seconds
Scrolling for: 16 seconds
Error: selenium.common.exceptions.WebDriverException: Message: unknown error: session deleted because of page crash
from unknown error: cannot determine loading status
from tab crashed
(Session info: chrome=113.0.5672.92)
Stacktrace:
#0 0x564cce0bd133 <unknown>
#1 0x564ccddf17f0 <unknown>
#2 0x564ccdddb0c2 <unknown>
#3 0x564ccddd9b18 <unknown>
#4 0x564ccddda2c2 <unknown>
#5 0x564ccdde7e8f <unknown>
#6 0x564ccdde8f32 <unknown>
#7 0x564ccddfb7fb <unknown>
#8 0x564ccde0003b <unknown>
#9 0x564ccddda863 <unknown>
#10 0x564ccddfb420 <unknown>
#11 0x564ccde660d4 <unknown>
#12 0x564ccde4d9a3 <unknown>
#13 0x564ccde2246a <unknown>
#14 0x564ccde2355e <unknown>
#15 0x564cce07ccae <unknown>
#16 0x564cce0808fe <unknown>
#17 0x564cce089f20 <unknown>
#18 0x564cce081923 <unknown>
#19 0x564cce054c0e <unknown>
#20 0x564cce0a4b08 <unknown>
#21 0x564cce0a4c97 <unknown>
#22 0x564cce0b5113 <unknown>
#23 0x7f33aaf8c6ea start_thread
Another is:
[0510/051204.886939:ERROR:file_io_posix.cc(152)] open /home/matt/.config/google-chrome/Crash Reports/pending/390c0413-938f-4885-a704-658eeb19a41b.lock: File exists (17)
[0510/051205.022187:ERROR:file_io_posix.cc(152)] open /home/matt/.config/google-chrome/Crash Reports/pending/390c0413-938f-4885-a704-658eeb19a41b.lock: File exists (17)
[0510/051205.373453:ERROR:file_io_posix.cc(152)] open /home/matt/.config/google-chrome/Crash Reports/pending/390c0413-938f-4885-a704-658eeb19a41b.lock: File exists (17)
[8884:9037:0510/051205.448374:ERROR:file_path_watcher_inotify.cc(315)] inotify_init() failed: Demasiados ficheros abiertos (24)
[8884:9040:0510/051205.467166:ERROR:platform_thread_posix.cc(155)] pthread_create: Recurso no disponible temporalmente (11)
[8884:9038:0510/051205.467403:ERROR:platform_thread_posix.cc(155)] pthread_create: Recurso no disponible temporalmente (11)
[8884:9037:0510/051205.467202:ERROR:platform_thread_posix.cc(155)] pthread_create: Recurso no disponible temporalmente (11)
[8884:8884:0510/051205.469232:ERROR:platform_thread_posix.cc(155)] pthread_create: Recurso no disponible temporalmente (11)
[8884:8884:0510/051205.469495:ERROR:platform_thread_posix.cc(155)] pthread_create: Recurso no disponible temporalmente (11)
[0510/051205.481063:ERROR:file_io_posix.cc(152)] open /home/matt/.config/google-chrome/Crash Reports/pending/390c0413-938f-4885-a704-658eeb19a41b.lock: File exists (17)
[0510/051205.489100:ERROR:nacl_helper_linux.cc(355)] NaCl helper process running without a sandbox!
Most likely you need to configure your SUID sandbox correctly
Error: selenium.common.exceptions.WebDriverException: Message: unknown error: cannot connect to chrome at 127.0.0.1:49429
from chrome not reachable
Stacktrace:
#0 0x55f886065133 <unknown>
#1 0x55f885d997f0 <unknown>
#2 0x55f885d87656 <unknown>
#3 0x55f885dc6f60 <unknown>
#4 0x55f885dbe932 <unknown>
#5 0x55f885dfe9ad <unknown>
#6 0x55f885dfe18f <unknown>
#7 0x55f885df59a3 <unknown>
#8 0x55f885dca46a <unknown>
#9 0x55f885dcb55e <unknown>
#10 0x55f886024cae <unknown>
#11 0x55f8860288fe <unknown>
#12 0x55f886031f20 <unknown>
#13 0x55f886029923 <unknown>
#14 0x55f885ffcc0e <unknown>
#15 0x55f88604cb08 <unknown>
#16 0x55f88604cc97 <unknown>
#17 0x55f88605d113 <unknown>
#18 0x7fef36c946ea start_thread
Another:
[0510/052704.045492:ERROR:file_io_posix.cc(152)] open /home/matt/.config/google-chrome/Crash Reports/pending/390c0413-938f-4885-a704-658eeb19a41b.lock: File exists (17)
[0510/052704.725063:ERROR:spawn_subprocess.cc(136)] fork: Resource temporarily unavailable (11)
[0510/052704.725454:FATAL:thread_posix.cc(29)] Check failed: . pthread_create: Resource temporarily unavailable (11)
[0510/052704.887124:ERROR:file_io_posix.cc(152)] open /home/matt/.config/google-chrome/Crash Reports/pending/390c0413-938f-4885-a704-658eeb19a41b.lock: File exists (17)
[0510/052705.022366:ERROR:file_io_posix.cc(152)] open /home/matt/.config/google-chrome/Crash Reports/pending/390c0413-938f-4885-a704-658eeb19a41b.lock: File exists (17)
[0510/053004.657512:ERROR:file_io_posix.cc(152)] open /home/matt/.config/google-chrome/Crash Reports/pending/390c0413-938f-4885-a704-658eeb19a41b.lock: File exists (17)
[0510/053005.847794:ERROR:file_io_posix.cc(152)] open /home/matt/.config/google-chrome/Crash Reports/pending/390c0413-938f-4885-a704-658eeb19a41b.lock: File exists (17)
/usr/lib64/R/bin/R: fork: retry: Recurso no disponible temporalmente
/usr/lib64/R/bin/R: fork: retry: Recurso no disponible temporalmente
/usr/lib64/R/bin/R: fork: retry: Recurso no disponible temporalmente
[0510/053303.999071:ERROR:file_io_posix.cc(152)] open /home/matt/.config/google-chrome/Crash Reports/pending/390c0413-938f-4885-a704-658eeb19a41b.lock: File exists (17)
/usr/lib64/R/bin/R: fork: retry: Recurso no disponible temporalmente
/usr/lib64/R/bin/R: fork: retry: Recurso no disponible temporalmente
/usr/lib64/R/bin/R: fork: retry: Recurso no disponible temporalmente
sh: fork: retry: Recurso no disponible temporalmente
sh: fork: retry: Recurso no disponible temporalmente
sh: fork: retry: Recurso no disponible temporalmente
sh: fork: retry: Recurso no disponible temporalmente
sh: fork: Recurso no disponible temporalmente
sh: fork: retry: Recurso no disponible temporalmente
sh: fork: retry: Recurso no disponible temporalmente
sh: fork: retry: Recurso no disponible temporalmente
Error in python_config_impl(python) :
Error 254 occurred running /home/matt/.asdf/shims/python3:
Calls: use_python -> python_config -> python_config_impl
Además: Warning message:
In system2(command = python, args = shQuote(script), stdout = TRUE, :
comando ejecutado ''/home/matt/.asdf/shims/python3' '/home/matt/R/x86_64-suse-linux-gnu-library/4.1/reticulate/config/config.py' 2>/dev/null' tiene estatus 254
Ejecución interrumpida
[0510/053353.142478:ERROR:file_io_posix.cc(152)] open /home/matt/.config/google-chrome/Crash Reports/pending/390c0413-938f-4885-a704-658eeb19a41b.lock: File exists (17)
[0510/053604.007883:ERROR:file_io_posix.cc(152)] open /home/matt/.config/google-chrome/Crash Reports/pending/390c0413-938f-4885-a704-658eeb19a41b.lock: File exists (17)
[0510/053604.254775:ERROR:file_io_posix.cc(152)] open /home/matt/.config/google-chrome/Crash Reports/pending/390c0413-938f-4885-a704-658eeb19a41b.lock: File exists (17)
[14020:14020:0510/053704.324600:ERROR:platform_thread_posix.cc(155)] pthread_create: Recurso no disponible temporalmente (11)
[0510/053704.350727:ERROR:file_io_posix.cc(152)] open /home/matt/.config/google-chrome/Crash Reports/pending/390c0413-938f-4885-a704-658eeb19a41b.lock: File exists (17)
[14023:14023:0510/053704.394923:ERROR:platform_thread_posix.cc(155)] pthread_create: Recurso no disponible temporalmente (11)
[0510/053704.402805:ERROR:file_io_posix.cc(152)] open /home/matt/.config/google-chrome/Crash Reports/pending/390c0413-938f-4885-a704-658eeb19a41b.lock: File exists (17)
[0510/053904.777602:ERROR:file_io_posix.cc(152)] open /home/matt/.config/google-chrome/Crash Reports/pending/390c0413-938f-4885-a704-658eeb19a41b.lock: File exists (17)
[0510/053904.874287:ERROR:file_io_posix.cc(152)] open /home/matt/.config/google-chrome/Crash Reports/pending/390c0413-938f-4885-a704-658eeb19a41b.lock: File exists (17)
....
Is there a way I can delete the files in the folder where the undetected chrome driver is located?
When I restart the computer the web-scraping code works fine again for a few hours.
