I'm still really new to Julia and am trying to find small things to do to help me learn but I am currently stuck. I want to create a web-scraper in Julia using Selenium. WebDriver.jl doesn't seem to have been updated to work with Julia 1.0+ and my attempts at using PyCall are being hindered by an error.
My code presently looks something like this:
using PyCall
@pyimport selenium
@pyimport selenium.webdriver as webdriver
profile = webdriver.FirefoxProfile()
profile[:set_preference]("browser.download.folderList", 2)
profile[:set_preference]("browser.download.manager.showWhenStarting", false)
profile[:set_preference]("browser.download.dir", cd())
profile[:set_preference]("browser.helperApps.neverAsk.saveToDisk",
"text/plain,text/x-csv,text/csv,application/vnd.ms-excel,application/csv,application/x-csv,text/csv,text/comma-separated-values,text/x-comma-separated-values,text/tab-separated-values,application/pdf")
options = webdriver.FirefoxOptions()
options[:set_headless](headless=false)
driver = webdriver.Firefox(firefox_profile=profile, options=options, executable_path="/home/mcamp/PythonProjects/BudgetApp/geckodriver")
The last line gives this error:
ERROR: PyError ($(Expr(:escape, :(ccall(#= /home/mcamp/.julia/packages/PyCall/0jMpb/src/pyfncall.jl:44 =# @pysym(:PyObject_Call), PyPtr, (PyPtr, PyPtr, PyPtr), o, pyargsptr, kw))))) <class 'selenium.common.exceptions.SessionNotCreatedException'>
SessionNotCreatedException('Failed to set preferences: Unable to read profile preferences file', None, None)
File "/home/mcamp/anaconda3/lib/python3.6/site-packages/selenium/webdriver/firefox/webdriver.py", line 174, in __init__
keep_alive=True)
File "/home/mcamp/anaconda3/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
self.start_session(capabilities, browser_profile)
File "/home/mcamp/anaconda3/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/home/mcamp/anaconda3/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/home/mcamp/anaconda3/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
Stacktrace:
[1] pyerr_check at /home/mcamp/.julia/packages/PyCall/0jMpb/src/exception.jl:60 [inlined]
[2] pyerr_check at /home/mcamp/.julia/packages/PyCall/0jMpb/src/exception.jl:64 [inlined]
[3] macro expansion at /home/mcamp/.julia/packages/PyCall/0jMpb/src/exception.jl:84 [inlined]
[4] __pycall!(::PyObject, ::Ptr{PyCall.PyObject_struct}, ::PyObject, ::PyObject) at /home/mcamp/.julia/packages/PyCall/0jMpb/src/pyfncall.jl:44
[5] _pycall!(::PyObject, ::PyObject, ::Tuple{}, ::Int64, ::PyObject) at /home/mcamp/.julia/packages/PyCall/0jMpb/src/pyfncall.jl:22
[6] _pycall!(::PyObject, ::PyObject, ::Tuple{}, ::Base.Iterators.Pairs{Symbol,Any,Tuple{Symbol,Symbol,Symbol},NamedTuple{(:firefox_profile, :options, :executable_path),Tuple{PyObject,Nothing,String}}}) at /home/mcamp/.julia/packages/PyCall/0jMpb/src/pyfncall.jl:11
[7] #call#89 at /home/mcamp/.julia/packages/PyCall/0jMpb/src/pyfncall.jl:89 [inlined]
[8] (::getfield(PyCall, Symbol("#kw#PyObject")))(::NamedTuple{(:firefox_profile, :options, :executable_path),Tuple{PyObject,Nothing,String}}, ::PyObject) at ./none:0
[9] top-level scope at none:0
I can get Firefox to launch and do some simple things by excluding the firefox_profile and options. What I think is happening is that I am trying to pass a Python Object to the method and Julia is probably wrapped around it mutating it so that Python isn't sure what to do with it.
TIA
