3

I have a python script calling the selenium webdriver for Firefox. That works fine if I call the Python script from CMD but not when called from a simple PHP page.

I have tried multiple things and simplified my scripts but no luck.

A similar script works fine if I host it on my local Windows server. However this does not work when hosted on AWS.

PHP script:

$user_name = 'test';
$short_lang = 'fr';

$pyscript = DIR_PYTHON_SCRIPTS.'retrieve_player_worlds.py';
$python = 'C:\Python39\python.exe';

$cmd = "$python $pyscript $user_name $short_lang";
exec("$cmd", $python_output);

return $python_output;

Extract of Python script:

#!C:/Python39/python.exe
import sys
import json

from seleniumwire import webdriver
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
from selenium.webdriver import Firefox
from selenium.webdriver.firefox.options import Options

opts = Options()
opts.log.level = "trace"

try:
    binary = FirefoxBinary(r'C:\\Program Files (x86)\\Mozilla Firefox\\Firefox.exe')
    driver = webdriver.Firefox(firefox_binary=binary, executable_path='C:\\xampp\\htdocs\\path\\scripts\\python\\geckodriver.exe', options=opts)
    driver.get('https://www.google.se')   # tried different URLs
    driver.maximize_window()

except Exception as e:
    logging.exception("Exception occurred binary firefox")

driver.quit()

geckodriver logs:

606076677289    geckodriver INFO    Listening on 127.0.0.1:50050
1606076680370   mozrunner::runner   INFO    Running command: "C:\\\\Program Files (x86)\\\\Mozilla Firefox\\\\Firefox.exe" "--marionette" "--headless" "-foreground" "-no-remote" "-profile" "C:\\Windows\\TEMP\\rust_mozprofileQQJMxG"
1606076680370   geckodriver::marionette DEBUG   Waiting 60s to connect to browser on 127.0.0.1:50058
*** You are running in headless mode.
1606076685899   Marionette  TRACE   Marionette enabled
1606076685955   Marionette  TRACE   Received observer notification toplevel-window-ready
JavaScript error: resource://gre/modules/XULStore.jsm, line 66: Error: Can't find profile directory.
console.warn: SearchSettings: "get: No settings file exists, new profile?" (new Error("", "(unknown module)"))
1606076688307   Marionette  TRACE   Received observer notification marionette-startup-requested
1606076688308   Marionette  TRACE   Waiting until startup recorder finished recording startup scripts...
1606076688323   Marionette  TRACE   All scripts recorded.
1606076688323   Marionette  DEBUG   Setting recommended pref apz.content_response_timeout to 60000
1606076688323   Marionette  DEBUG   Setting recommended pref browser.contentblocking.introCount to 99
1606076688323   Marionette  DEBUG   Setting recommended pref browser.download.panel.shown to true
1606076688323   Marionette  DEBUG   Setting recommended pref browser.newtabpage.enabled to false
1606076688324   Marionette  DEBUG   Setting recommended pref browser.pagethumbnails.capturing_disabled to true
1606076688325   Marionette  DEBUG   Setting recommended pref browser.search.update to false
1606076688325   Marionette  DEBUG   Setting recommended pref browser.tabs.disableBackgroundZombification to false
1606076688325   Marionette  DEBUG   Setting recommended pref browser.tabs.remote.separatePrivilegedContentProcess to false
1606076688325   Marionette  DEBUG   Setting recommended pref browser.tabs.unloadOnLowMemory to false
1606076688325   Marionette  DEBUG   Setting recommended pref browser.tabs.warnOnCloseOtherTabs to false
1606076688325   Marionette  DEBUG   Setting recommended pref browser.tabs.warnOnOpen to false
1606076688325   Marionette  DEBUG   Setting recommended pref browser.usedOnWindows10.introURL to 
1606076688325   Marionette  DEBUG   Setting recommended pref browser.urlbar.suggest.searches to false
1606076688326   Marionette  DEBUG   Setting recommended pref datareporting.policy.dataSubmissionPolicyAccepted to false
1606076688326   Marionette  DEBUG   Setting recommended pref dom.disable_beforeunload to true
1606076688326   Marionette  DEBUG   Setting recommended pref dom.disable_open_during_load to false
1606076688326   Marionette  DEBUG   Setting recommended pref dom.file.createInChild to true
1606076688326   Marionette  DEBUG   Setting recommended pref dom.max_chrome_script_run_time to 0
1606076688326   Marionette  DEBUG   Setting recommended pref dom.max_script_run_time to 0
1606076688327   Marionette  DEBUG   Setting recommended pref dom.push.connection.enabled to false
1606076688327   Marionette  DEBUG   Setting recommended pref extensions.getAddons.cache.enabled to false
1606076688327   Marionette  DEBUG   Setting recommended pref extensions.getAddons.discovery.api_url to data:, 
1606076688327   Marionette  DEBUG   Setting recommended pref network.http.prompt-temp-redirect to false
1606076688327   Marionette  DEBUG   Setting recommended pref privacy.trackingprotection.enabled to false
1606076688328   Marionette  DEBUG   Setting recommended pref security.fileuri.strict_origin_policy to false
1606076688328   Marionette  DEBUG   Setting recommended pref security.notification_enable_delay to 0
1606076688328   Marionette  DEBUG   Setting recommended pref signon.autofillForms to false
1606076688329   Marionette  DEBUG   Setting recommended pref signon.rememberSignons to false
1606076688332   Marionette  INFO    Listening on port 50058
1606076688333   Marionette  DEBUG   Marionette is listening
1606076688806   Marionette  DEBUG   Accepted connection 0 from 127.0.0.1:50080
1606076688798   geckodriver::marionette DEBUG   Connection to Marionette established on 127.0.0.1:50058.
1606076688822   Marionette  DEBUG   0 -> [0,1,"WebDriver:NewSession",{"acceptInsecureCerts":true,"browserName":"firefox","proxy":{"httpProxy":"127.0.0.1:50048","noProxy":[],"proxyType":"manual","sslProxy":"127.0.0.1:50048"}}]
1606076688823   Marionette  WARN    TLS certificate errors will be ignored for this session
1606076688826   Marionette  INFO    Proxy settings initialised: {"proxyType":"manual","httpProxy":"127.0.0.1:50048","noProxy":[],"sslProxy":"127.0.0.1:50048"}
1606076688837   Marionette  TRACE   [21] Frame script loaded
1606076688886   Marionette  DEBUG   0 <- [1,1,null,{"sessionId":"f9a971f5-6890-48f2-b7ad-c0d63583e907","capabilities":{"browserName":"firefox","browserVersion":"83.0" ... \rust_mozprofileQQJMxG","moz:shutdownTimeout":60000,"moz:useNonSpecCompliantPointerOrigin":false,"moz:webdriverClick":true}}]
1606076688892   webdriver::server   DEBUG   <- 200 OK {"value":{"sessionId":"f9a971f5-6890-48f2-b7ad-c0d63583e907","capabilities":{"acceptInsecureCerts":true,"browserName":"firefox","browserVersion":"83.0","moz:accessibilityChecks":false,"moz:buildID":"20201112153044","moz:geckodriverVersion":"0.28.0","moz:headless":true,"moz:processID":7724,"moz:profile":"C:\\Windows\\Temp\\rust_mozprofileQQJMxG","moz:shutdownTimeout":60000,"moz:useNonSpecCompliantPointerOrigin":false,"moz:webdriverClick":true,"pageLoadStrategy":"normal","platformName":"windows","platformVersion":"10.0","proxy":{"httpProxy":"127.0.0.1:50048","noProxy":[],"proxyType":"manual","sslProxy":"127.0.0.1:50048"},"rotatable":false,"setWindowRect":true,"strictFileInteractability":false,"timeouts":{"implicit":0,"pageLoad":300000,"script":30000},"unhandledPromptBehavior":"dismiss and notify"}}}
1606076688892   webdriver::server   DEBUG   -> POST /session/f9a971f5-6890-48f2-b7ad-c0d63583e907/url {"url": "https://www.google.se"}
1606076688902   Marionette  DEBUG   0 -> [0,2,"WebDriver:Navigate",{"url":"https://www.google.se"}]
1606076688916   Marionette  TRACE   Received message beforeunload for about:blank
1606076691334   Marionette  TRACE   Received message beforeunload for about:blank
1606076691341   Marionette  TRACE   Received message pagehide for about:blank
1606076691406   Marionette  TRACE   Received message DOMContentLoaded for about:neterror?e=nssFailure2&u=https%3A//www.google.se/&c=UTF-8&d=Anslutningen%20till%20www.google.se%20avbr%C3%B6ts%20under%20h%C3%A4mtningen%20av%20sidan.
1606076691407   Marionette  DEBUG   0 <- [1,2,{"error":"unknown error","message":"Reached error page: about:neterror?e=nssFailure2&u=https%3A//www.google.se/&c=UTF-8& ... adyState@chrome://marionette/content/navigate.js:60:24\nonNavigation@chrome://marionette/content/navigate.js:285:39\n"},null]
1606076691409   webdriver::server   DEBUG   <- 500 Internal Server Error {"value":{"error":"unknown error","message":"Reached error page: about:neterror?e=nssFailure2&u=https%3A//www.google.se/&c=UTF-8&d=Anslutningen%20till%20www.google.se%20avbr%C3%B6ts%20under%20h%C3%A4mtningen%20av%20sidan.","stacktrace":"WebDriverError@chrome://marionette/content/error.js:181:5\nUnknownError@chrome://marionette/content/error.js:488:5\ncheckReadyState@chrome://marionette/content/navigate.js:60:24\nonNavigation@chrome://marionette/content/navigate.js:285:39\n"}}                                                                                                                                                                                                                                        
1606076691440   Marionette  DEBUG   Closed connection 0

I have Googled the errors for a few days now but nothing has helped. I seems that the connection has been aborted at some point.

geckodriver.exe is in the right folder (C:\\xampp\\htdocs\\path\\scripts\\python\\) and permissions seem to be okay as the log file is created by the script. I'm using the latest version of Firefox, webdriver (0.28) and Selenium.

Additional information (as requested in comments below): After calling the webdriver, I interact with elements in the page and enter credentials:

iframe = driver.find_element_by_tag_name('iframe')
iframe_switched = driver.switch_to.frame(iframe)
    
useridInput = driver.find_element_by_id('login_userid')
useridInput.click();
    
useridInput.send_keys(user_name);

Update 2:

I compared the geckodriver logs when I run from CMD and from the PHP page. The difference is only in the end, everything else (including the command) is the same:

1606076691406   Marionette  TRACE   Received message DOMContentLoaded for about:neterror?e=nssFailure2&u=https%3A//www.google.se/&c=UTF-8&d=Anslutningen%20till%20www.google.se%20avbr%C3%B6ts%20under%20h%C3%A4mtningen%20av%20sidan.
1606076691407   Marionette  DEBUG   0 <- [1,2,{"error":"unknown error","message":"Reached error page: about:neterror?e=nssFailure2&u=https%3A//www.google.se/&c=UTF-8& ... adyState@chrome://marionette/content/navigate.js:60:24\nonNavigation@chrome://marionette/content/navigate.js:285:39\n"},null]
1606076691409   webdriver::server   DEBUG   <- 500 Internal Server Error {"value":{"error":"unknown error","message":"Reached error page: about:neterror?e=nssFailure2&u=https%3A//www.google.se/&c=UTF-8&d=Anslutningen%20till%20www.google.se%20avbr%C3%B6ts%20under%20h%C3%A4mtningen%20av%20sidan.","stacktrace":"WebDriverError@chrome://marionette/content/error.js:181:5\nUnknownError@chrome://marionette/content/error.js:488:5\ncheckReadyState@chrome://marionette/content/navigate.js:60:24\nonNavigation@chrome://marionette/content/navigate.js:285:39\n"}}                                                                                                                                                                                              
1606076691440   Marionette  DEBUG   Closed connection 0

Decoded and translated, this means: The connection to www.google.com was terminated while downloading the page

remyremy
  • 3,548
  • 3
  • 39
  • 56
  • What exactly do you do after calling the web driver? Just keep opening google?`404 Not Found {"value":{"error":"no such element","message":"Unable to locate element: iframe"` This error indicates as if you are trying to scrape something. Can you give some detail? – Abhishek Rai Nov 22 '20 at 13:23
  • I edited my question with the lines after the webdriver call. I basically interact with some elements to login into a page. I believe the problem is before that, as I don't even see Firefox opening when calling the Python script from PHP. It however opens when called directly from CMD, and the login page works fine. – remyremy Nov 22 '20 at 19:28
  • My Internet Options are set to "Automatically detect settings". I've tried with Firefox set to "Use system settings", "Auto-detect proxy settings" and "no proxy". Same thing... Note: The proxytype stays "Manual" whatever I choose (tried to restart Firefox in-between) – remyremy Nov 22 '20 at 19:44

1 Answers1

0

After a month of troubleshooting I finally found the problem. It seems that Firefox won't open if Apache runs as a Windows Service. If I run Apache as a standard app, it will open fine.

I've tried to make the Apache service run as the Local System Account with and without allowing interaction with Desktop, and as an Admin account but that does not help.

So in the end I uninstalled the Apache service and run it as an app (using Xampp)...

remyremy
  • 3,548
  • 3
  • 39
  • 56