Clicking dropdown option, getting base WebDriverException with empty Message

Question

Using Python & Selenium in Safari browser; trying to select from a dropdown box.

The dropdown looks like this in the HTML:

<select name="ctl00$cph1$d1$cboExchange" onchange="javascript:setTimeout('__doPostBack(\'ctl00$cph1$d1$cboExchange\',\'\')', 0)" id="ctl00_cph1_d1_cboExchange" style="width:205px;margin-left:30px;">
                <option value="AMEX">American Stock Exchange</option>
                <option value="ASX">Australian Securities Exchange</option>
                <option value="CFE">Chicago Futures Exchange</option>
                <option value="EUREX">EUREX Futures Exchange</option>
                <option value="FOREX">Foreign Exchange</option>
                <option selected="selected" value="INDEX">Global Indices</option>
                <option value="HKEX">Hong Kong Stock Exchange</option>
                <option value="KCBT">Kansas City Board of Trade</option>
                <option value="LIFFE">LIFFE Futures and Options</option>
                <option value="LSE">London Stock Exchange</option>
                <option value="MGEX">Minneapolis Grain Exchange</option>
                <option value="USMF">Mutual Funds</option>
                <option value="NASDAQ">NASDAQ Stock Exchange</option>
                <option value="NYBOT">New York Board of Trade</option>
                <option value="NYSE">New York Stock Exchange</option>
                <option value="OTCBB">OTC Bulletin Board</option>
                <option value="SGX">Singapore Stock Exchange</option>
                <option value="TSX">Toronto Stock Exchange</option>
                <option value="TSXV">Toronto Venture Exchange</option>
                <option value="WCE">Winnipeg Commodity Exchange</option>

            </select>

My select-related code looks like this:

    try:
        exchange_dropdown_id = 'ctl00_cph1_d1_cboExchange'

        exchange_dropdown = driver.find_element(by=By.ID, value=exchange_dropdown_id)
        exchange_dropdown_select = Select(exchange_dropdown)

        print('-- clicking dropdown element')
        exchange_dropdown_select.select_by_value('AMEX')

        print('IT WORKED!')

    except (selenium.common.exceptions.ElementClickInterceptedException,
            selenium.common.exceptions.ElementNotInteractableException,
            selenium.common.exceptions.ElementNotSelectableException,
            selenium.common.exceptions.ElementNotVisibleException,
            selenium.common.exceptions.ImeActivationFailedException,
            selenium.common.exceptions.ImeNotAvailableException,
            selenium.common.exceptions.InsecureCertificateException,
            selenium.common.exceptions.InvalidCookieDomainException,
            selenium.common.exceptions.InvalidCoordinatesException,
            selenium.common.exceptions.InvalidElementStateException,
            selenium.common.exceptions.InvalidSelectorException,
            selenium.common.exceptions.InvalidSessionIdException,
            selenium.common.exceptions.InvalidSwitchToTargetException,
            selenium.common.exceptions.JavascriptException,
            selenium.common.exceptions.MoveTargetOutOfBoundsException,
            selenium.common.exceptions.NoAlertPresentException,
            selenium.common.exceptions.NoSuchAttributeException,
            selenium.common.exceptions.NoSuchCookieException,
            selenium.common.exceptions.NoSuchElementException,
            selenium.common.exceptions.NoSuchFrameException,
            selenium.common.exceptions.NoSuchShadowRootException,
            selenium.common.exceptions.NoSuchWindowException,
            selenium.common.exceptions.ScreenshotException,
            selenium.common.exceptions.SeleniumManagerException,
            selenium.common.exceptions.SessionNotCreatedException,
            selenium.common.exceptions.StaleElementReferenceException,
            selenium.common.exceptions.TimeoutException,
            selenium.common.exceptions.UnableToSetCookieException,
            selenium.common.exceptions.UnexpectedAlertPresentException,
            selenium.common.exceptions.UnexpectedTagNameException,
            selenium.common.exceptions.UnknownMethodException,
            ) as ex:
        print('*** POSSIBLE EXCEPTION FOUND:\n' + repr(ex) + '\n*** END REPR')

    except selenium.common.exceptions.WebDriverException as ex:
        print('*** WEBDRIVER BASE EXCEPTION:\n' + repr(ex) + '\n*** END BASE EXCEPTION DETAIL')

    except Exception as ex:
        print('*** Failure selecting item. Exception:\n' + str(ex) + '*** END EXCEPTION MESSAGE ***')
        print('*** Exception Detail?:\n' + repr(ex) + '\n*** END EXCEPTION MESSAGE ***')

The result is:

-- clicking dropdown element
*** WEBDRIVER BASE EXCEPTION:
WebDriverException()
*** END BASE EXCEPTION DETAIL

If I use str(ex) instead of repr(ex), I get the word MESSAGE: with no other information.

I entered all the exceptions in the WebDriver API in hopes of catching "the one" that would tell me what's happening.

I have tried using select_by_visible_text('American Stock Exchange') as well, with the same result.

I can iterate through all the options with the Select object:

        for option in exchange_dropdown_select.options:
            print(option.get_attribute('value'))
            if option.get_attribute('value') == 'AMEX':
                print('AMEX found')

and it finds the option. However, if I add a click on the option:

        for option in exchange_dropdown_select.options:
            print(option.get_attribute('value'))
            if option.get_attribute('value') == 'AMEX':
                print('AMEX found')
                option.click()
                print('clicked AMEX')

it still ends with the empty WebDriver exception:

AMEX found
*** WEBDRIVER BASE EXCEPTION:
WebDriverException()
*** END BASE EXCEPTION DETAIL.

The HTML is enclosed in a table, if that makes a difference. And, I'm using Python 3.9.

Any ideas what might be happening and how I might fix it?

Also, I'd prefer not to use the XPATH or index values as the developers might change the order of the options, and the XPATH doesn't contain any ID information relating to the item(s) I want to select - again, a path that could change if the option order changes.

Oh, and my imports are:

import sys

import selenium # for all of those exceptions down there; will be removed when solved
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import Select
from selenium.webdriver.support.ui import WebDriverWait

Update: to get to the issue -

start with the URL eoddata.com
log in (free account is fine)
click the Download button at the upper left (next to Home Page button)
click the 'X' to close the popup

Then, I'm looking at the Download Data portion in the upper left.

In this particular example, I want to select the Exchange - value = "AMEX" and visible text = "American Stock Exchange". It's currently the first index, but again, this could change at any time, so I don't want to count on an exact position, if I can help it.

Finally, in respectful netizenship, the current robots.txt is:

User-agent: *
Crawl-delay:10
Disallow: /images/
Disallow: /styles/
Disallow: /scripts/

User-agent: turnitinbot 
Disallow: /

Sitemap: http://www.eoddata.com/sitemapindex.xml

Update 2: Apple Docs related to safaridriver (accessed 28 Feb 2023):

I have found a number of posts, without selected answers, asking about this issue in Safari. I've not found any instructional info with a SELECT specifically - and all of the rest of the steps I listed above are working perfectly.

It looks like I need a REST way of accessing the SELECT/OPTION item -- or, I need to switch to Firefox or Chrome, lol.

@soundwave, I've updated my post with the URL, steps to arrive at the code, and robots.txt showing 'bot' permissions — leanne, Feb 28 '23 at 17:48
To me it works without problems. Add your full python code in the question, and also the selenium version — sound wave, Feb 28 '23 at 18:33
To me it works without problems with chromedriver. Add your full python code in the question, and also the selenium version — sound wave, Feb 28 '23 at 18:39
I'm using Safari, whose driver is built-in and doesn't require an import. I've listed all my imports and the exact code for this step in my question above - here's my driver declaration: `driver = webdriver.Safari()`. All the previous steps (as listed in my update) worked as expected. It's only the dropdown's option select step that's failing. I am able to make it work via `driver.execute_script`; however, that requires the `XPATH`, which I was trying to avoid. Perhaps it's something with the Safari driver... — leanne, Feb 28 '23 at 19:06
Did you try with chrome instead of safari? What did you put inside `driver.execute_script()` to select the option? — sound wave, Feb 28 '23 at 19:16
Found [these details About WebDriver for Safari](https://developer.apple.com/documentation/webkit/about_webdriver_for_safari) on Apple's developer site. Apparently, it has extra security features to safeguard data. It could be that those features are blocking something. — leanne, Feb 28 '23 at 19:17
amex_xpath =\ '/html/body/form/div[3]/div[3]/div[1]/div/div/div[1]/div[2]/div[3]/table/tbody/tr/td[2]/select/option[1]' | option = exchange_dropdown.find_element(by=By.XPATH, value=amex_xpath) | driver.execute_script("arguments[0].click();", option) — leanne, Feb 28 '23 at 19:18
Is `exchange_dropdown.click()` missing? Because if the dropdown is closed, how can you click on the option? — sound wave, Feb 28 '23 at 19:23
I updated the answer with two new codes, one using `send_keys` and one using `click`, try and let me know — sound wave, Feb 28 '23 at 19:31

sound wave · Accepted Answer · 2023-02-28T20:20:07.213

0

To me this works without problems using chromedriver

from selenium.webdriver.support.ui import Select
exchange_dropdown = driver.find_element(By.ID, 'ctl00_cph1_d1_cboExchange')
Select(exchange_dropdown).select_by_value('AMEX')

Alternative (1)

exchange_dropdown = driver.find_element(By.ID, 'ctl00_cph1_d1_cboExchange')
option = exchange_dropdown.find_element(By.CSS_SELECTOR, 'option[value=AMEX]')
exchange_dropdown.send_keys(option.text)

Alternative (2)

exchange_dropdown = driver.find_element(By.ID, 'ctl00_cph1_d1_cboExchange')
option = exchange_dropdown.find_element(By.CSS_SELECTOR, 'option[value=AMEX]')
option.click()

edited Feb 28 '23 at 20:20

answered Feb 28 '23 at 09:13

sound wave

3,191
3
11
29

The traceback was empty as well. Put in: `print('And the traceback:\n') traceback.print_exc() print('END OF TRACEBACK')` Got out: *** WEBDRIVER BASE EXCEPTION: WebDriverException() *** END BASE EXCEPTION DETAIL And the traceback: END OF TRACEBACK" -- I put the `import` at top – leanne Feb 28 '23 at 18:22
Sorry, By.ID and By.CSS_SELECTOR not working in Safari. Thanks for the time, though. I have found multiple posts, without answers, having difficulty with the SELECT/OPTIONS thing, so again, it might be something specific to the Safari WebDriver. – leanne Feb 28 '23 at 19:54
@leanne How is it possible that By.ID is not working? You used it yourself in the question `exchange_dropdown = driver.find_element(by=By.ID, value=exchange_dropdown_id)` – sound wave Feb 28 '23 at 20:05
Sorry for the confusion - those are working fine; the `.click()` following them is not. – leanne Feb 28 '23 at 20:13
@leanne Try to replace `option.click()` with `driver.execute_script("arguments[0].click()", option)`. Moreover, did you try the alternative using `send_keys`? – sound wave Feb 28 '23 at 20:16
Okay... after lunch and a breather... Alternative(1), using `send_keys` seems to be working - it did not give me an exception. Again, thanks very much for your time. – leanne Feb 28 '23 at 21:01
@leanne Good! Luckily there are many alternatives, so you can keep safari... for now :P – sound wave Mar 01 '23 at 08:00

Clicking dropdown option, getting base WebDriverException with empty Message

1 Answers1