Using ChromeDriver v102 (since there's a current bug open for v103 https://bugs.chromium.org/p/chromedriver/issues/detail?id=4121)
I get the error (full log information listed below):
Response code 500. Message: unknown error: cannot create default profile directory
when I run ChromeDriver remotely on the selenium node grid.
I am able to use a specific chrome profile when running the ChromeDriver locally. The driver for Firefox also works remotely, so I have to conclude that this is a "remote ChromeDriver" issue.
I download the chrome profile from the cloud into my current working directory. The directory is detected/located in my code.
There are no spaces in my path to the profile directory. The chrome profile only contains essential files I want to keep.
save_file_list = ["Bookmarks", "History", "History-journal", "Cookies", "Cookies-Journal", "Favicons", "Favicons-journal"]
When I use the ChromeDriver locally, I download the chrome profile, it loads the essential files and the driver creates the rest on its own. The same code works (and runs in an executable) when I run it on windows and mac machines.
However, this fails to work at all when using the remote driver. The remote driver works perfectly when I do not use a specific user-data-dir. It even works using an authenticated proxy. Therefore there are no issues with my connection. The code fails in both cases: when I use a virtual environment to run the code and when I run a docker image. Therefore I know it's not a docker issue.
This is the code for loading a chrome profile to my chrome options:
def add_profile(self, chrome_options):
if self.download_profile():
arg = f'--user-data-dir={os.getcwd()}/{self.profile_path}'
logging.info(f'Adding arg: {arg}')
chrome_options.add_argument(arg)
logging.info("Successfully added profile to chrome driver.")
return chrome_options
def setup_chrome(self) -> webdriver.Remote:
prefs = {
"credentials_enable_service": False,
"profile.password_manager_enabled": False,
}
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--enable-javascript")
chrome_options.add_argument("--disable-blink-features=AutomationControlled")
chrome_options.add_experimental_option("useAutomationExtension", False)
chrome_options.add_experimental_option("excludeSwitches", ["enable-automation"])
chrome_options.add_experimental_option("prefs", prefs)
chrome_options = self.proxy.add_proxy_plugin(chrome_options, use_proxy=True)
chrome_options = self.chrome_profile.add_profile(chrome_options) #currently not working
capabilities = {
"browserName": "chrome",
"chromeOptions": {
"useAutomationExtension": False,
"forceDevToolsScreenshot": True,
"args": ["--start-maximized", "--disable-infobars"],
},
}
remote_driver_host = os.environ.get("SELENIUM_URL")
logging.info("Setting up remote connection")
try:
selenium_connection = RemoteConnectionV2(
remote_driver_host, keep_alive=False
)
selenium_connection.set_remote_connection_authentication_headers()
driver = webdriver.Remote(
command_executor=selenium_connection, # for prod
# command_executor="http://localhost:4444/wd/hub", # for local development
desired_capabilities=capabilities,
options=chrome_options,
keep_alive=True,
)
logging.info("SUCCESS: created remote chrome using connection")
except Exception as ex:
logging.error(f"Failed to initiate Chrome browser: {ex}")
raise
try:
driver.get("https://www.google.com")
except Exception as e:
logging.error(f"Failed to connect using proxy: {e}")
return driver
This is the log:
27-Jun-22 10:04:53 AM EDT | INFO | SUCCESS: downloaded proxy assignment:
27-Jun-22 10:04:54 AM EDT | INFO | Successfully downloaded & extracted _zGF_23_CHROME_PROFILE.
27-Jun-22 10:04:54 AM EDT | INFO | Adding arg: --user-data-dir=/Users/<myuser>/Repos<my-repo>/_zGF_23_CHROME_PROFILE
27-Jun-22 10:04:54 AM EDT | INFO | Successfully added profile to chrome driver.
27-Jun-22 10:04:54 AM EDT | INFO | Setting up remote connection
[Authentication] No identity token was found in the environment. Requesting a new one.
[Authentication] An identity token found successfully.
27-Jun-22 10:04:55 AM EDT | ERROR | Failed to initiate Chrome browser: Message: Could not start a new session. Error while creating session with the driver service. Stopping driver service: Could not start a new session. Response code 500. Message: unknown error: cannot create default profile directory
Build info: version: '4.2.2', revision: '683ccb65d6'
System info: host: 'localhost', ip: '127.0.0.1', os.name: 'Linux', os.arch: 'amd64', os.version: '4.4.0', java.version: '11.0.15'
Driver info: driver.version: unknown
Build info: version: '4.2.2', revision: '683ccb65d6'
System info: host: 'localhost', ip: '127.0.0.1', os.name: 'Linux', os.arch: 'amd64', os.version: '4.4.0', java.version: '11.0.15'
Driver info: driver.version: unknown
Stacktrace:
at org.openqa.selenium.grid.node.config.DriverServiceSessionFactory.apply (DriverServiceSessionFactory.java:194)
at org.openqa.selenium.grid.node.config.DriverServiceSessionFactory.apply (DriverServiceSessionFactory.java:67)
at org.openqa.selenium.grid.node.local.SessionSlot.apply (SessionSlot.java:145)
at org.openqa.selenium.grid.node.local.LocalNode.newSession (LocalNode.java:362)
at org.openqa.selenium.grid.distributor.local.LocalDistributor.startSession (LocalDistributor.java:618)
at org.openqa.selenium.grid.distributor.local.LocalDistributor.newSession (LocalDistributor.java:544)
at org.openqa.selenium.grid.distributor.local.LocalDistributor$NewSessionRunnable.handleNewSessionRequest (LocalDistributor.java:791)
at org.openqa.selenium.grid.distributor.local.LocalDistributor$NewSessionRunnable.lambda$run$1 (LocalDistributor.java:752)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:628)
at java.lang.Thread.run (Thread.java:829)
27-Jun-22 10:04:55 AM EDT | ERROR | Message: Could not start a new session. Error while creating session with the driver service. Stopping driver service: Could not start a new session. Response code 500. Message: unknown error: cannot create default profile directory
Build info: version: '4.2.2', revision: '683ccb65d6'
System info: host: 'localhost', ip: '127.0.0.1', os.name: 'Linux', os.arch: 'amd64', os.version: '4.4.0', java.version: '11.0.15'
Driver info: driver.version: unknown
Build info: version: '4.2.2', revision: '683ccb65d6'
System info: host: 'localhost', ip: '127.0.0.1', os.name: 'Linux', os.arch: 'amd64', os.version: '4.4.0', java.version: '11.0.15'
Driver info: driver.version: unknown
Stacktrace:
at org.openqa.selenium.grid.node.config.DriverServiceSessionFactory.apply (DriverServiceSessionFactory.java:194)
at org.openqa.selenium.grid.node.config.DriverServiceSessionFactory.apply (DriverServiceSessionFactory.java:67)
at org.openqa.selenium.grid.node.local.SessionSlot.apply (SessionSlot.java:145)
at org.openqa.selenium.grid.node.local.LocalNode.newSession (LocalNode.java:362)
at org.openqa.selenium.grid.distributor.local.LocalDistributor.startSession (LocalDistributor.java:618)
at org.openqa.selenium.grid.distributor.local.LocalDistributor.newSession (LocalDistributor.java:544)
at org.openqa.selenium.grid.distributor.local.LocalDistributor$NewSessionRunnable.handleNewSessionRequest (LocalDistributor.java:791)
at org.openqa.selenium.grid.distributor.local.LocalDistributor$NewSessionRunnable.lambda$run$1 (LocalDistributor.java:752)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:628)
at java.lang.Thread.run (Thread.java:829)
