0

I am iterating through thousands of domains and using Selenium to visit all of them in a Windows Virtual Machine. My issue is that the driver I am using (undetected-chromedriver) keeps getting randomly stuck after a few hundred/thousand pages. It does not trigger my set timeout and stays like that for like 10-20 minutes which then throws an exception like Timed out receiving message from renderer: -1062.446 (why is it negative?) and continues on with my list.

The last step I see in my logs is a POST request being sent. Since I am running in non-headless mode, I see the site being loaded fine in the browser while the script is seemingly frozen. If I rerun my script and put the page it got stuck on to the beginning of the list, it then executes fine without issues.

Any ideas what could be causing the issue?

My driver setup:

import undetected_chromedriver as uc

def setup_chromium_browser(self):
        self._selenium_options = uc.ChromeOptions()

        self._selenium_options.add_argument("--disable-renderer-backgrounding")
        self._selenium_options.add_argument("--disable-backgrounding-occluded-windows")
        
        self.driver = uc.Chrome(
            options=self._selenium_options
        )

        self.driver.set_page_load_timeout(60)
        self.driver.set_script_timeout(60)

Note: Found the exact same issue but no solution was found. Selenium webdriver keeps getting stuck at random times

clueless_robot
  • 3
  • 2
  • I would give it a go without setting any options, just to rule those out. (timeouts and backgrounding flags) If you can, give the VM more memory. VM's and browsers are memory hungry... another thought is a site checking the visibility api after a pupunder. https://developer.mozilla.org/en-US/docs/Web/API/Page_Visibility_API There's a workaround on this site that involves firing the event if that's the case. – pcalkins Nov 08 '22 at 18:11
  • What is weird that the 60 second page timeout works for most pages and logs show `Timed out receiving message from renderer: ` have a positive <60 value, while the sites that hang the script always have a negative timeout value like -1062.446 – clueless_robot Nov 09 '22 at 10:31

0 Answers0