Cant interact with iframe after switching to it Python/Selenium-wire

Question

I'm trying to interact with a button on a page with the following structure. The button of interest is within a div in the body of an iframe, which is inside the main body. I've already read all the StackOverflow questions about how to switch to an iframe - as you can see below I have no issue with that. What I have an issue with is that regardless of that advice I am unable to interact with the iframe I switched to. The goal is to click a button inside a specific iframe.

EDIT: The issue seems to be related to the selenium-wire library. Here is the full code.

from seleniumwire import webdriver
from selenium.webdriver.chrome.options import Options
from webdriver_manager.chrome import ChromeDriverManager
import time

url = 'https://sahkovertailu.fi/'
options = {
    'suppress_connection_errors': False,
    'connection_timeout': None# Show full tracebacks for any connection errors
}
chrome_options = Options()
chrome_options.add_argument("--start-maximized")
chrome_options.add_argument('--auto-open-devtools-for-tabs')
chrome_options.add_argument('--log-level=2')

driver = webdriver.Chrome(ChromeDriverManager().install(), seleniumwire_options=options, chrome_options=chrome_options)

if driver.requests is not None:
    del driver.requests
driver.get(url)

iframe = driver.find_element_by_xpath("//iframe[contains(@src, 'privacy')]")
driver.switch_to.frame(iframe)
iframeUniqueId = iframe.get_attribute('cd_frame_id_')
print(iframeUniqueId)
time.sleep(2)
button = driver.find_element_by_xpath("//button[contains(., 'OK')]")
button.click()

Here is an example of the page layout

<!doctype html>
<html ng-app="someApp" xmlns="http://www.w3.org/1999/html">
    <head>
        <script>a lot of scripts</script>
    </head>
    <body class="unwantedBody">
        <iframe> some iframes</iframe>
        <div> different divs </div>
        <main> 
            some content
            <iframe> multiple iframes on different nested levels </iframe>
        </main>
        <div> more divs </div>
        <script> more scripts </script>
        <div id='interesting div'>
            <iframe src="uniqueString">
                <!doctype html>
                #document
                <html>
                    <body>
                        <div>
                            <button id='bestButton'>
                                'Nice title'
                            </button>
                        </div>
                    </body>
                </html>
            </iframe>
        </div>
    </body>
</html>

Using Jupyter Notebook I've been able to locate the iframe and switch to it. The problem is not related to trying to interact with the iFrame too fast, because I control the speed. I've tried using Expected conditions and waiting until the iframe can be switched to, but it is irrelevant to my problem.

driver.switch_to.default_content # Making sure no other frame is selected
iframe = driver.find_element_by_xpath("//iframe[contains(@src, 'uniqueString')]")
driver.switch_to.frame(iframe)
print(iframe.get_attribute('id'))

The above code prints out "interesting div", so it successfully finds the div where the iframe is and apparently selects the div? When I switch to the iframe, the webpage changes and an unique GUID is given to the iframe:

<iframe src="uniqueString" cd_frame_id_ = "uniqueGuidThatAlwaysChanges"> ... </iframe>

I am able to access the unique id with

iframe.get_attribute('cd_frame_id_')

Then I try to parse the iframe like this:

bestButton = driver.find_element_by_xpath("//button[@id = 'bestButton']")
bestButton.click()

This gives the error:

Message: element not interactable

If I replace the above driver. with iframe.find_element_by_xpath... I get the following error:

Message: no such element: Unable to locate element: {"method":"xpath","selector":"//button[@id="bestButton"]"}

I also tried to interact with the body within the iframe after switching to it with the above switch_to.frame(iframe), so in this example driver is already at the iframe:

document = driver.find_element_by_xpath('//html/body')
info = document.get_attribute('class')
print(info)

This prints out

unwantedBody

So somehow the driver has not switched to the iFrame I specified and instead is still stuck on the main HTML. When loading the webpage on chrome I can find the button I want with just this XPath //button[contains(@id='bestButton')] but in Selenium it doesn't work, because it is split by the #document within the iframe.

What am I missing? If it helps, the iFrame I am interested in is actually a modal window about cookie consent, which I am trying to get rid of to interact with the site.

Do you mind sharing the url if it's public ? – Akshay Aug 15 '20 at 07:43 — Akshay, Aug 15 '20 at 07:43
https://sahkovertailu.fi/ @Akshay – qoob Aug 15 '20 at 09:22 — qoob, Aug 15 '20 at 09:22

score 1 · Accepted Answer · answered Aug 15 '20 at 19:54

@qoob, you didn't specified where exactly is the button located on the website and Finnish to English translation was not that good by google. So I assumed you are talking about the first page itself as the DOM structure of the iframe looked familiar to what you have asked in the question. If the button in the below image is the actual point of interest then below code works.

from selenium import webdriver
import time

driver = webdriver.Chrome(r"C:\chromedriver.exe")
driver.maximize_window()
driver.get("https://sahkovertailu.fi/")
driver.switch_to.frame(driver.find_element_by_xpath("//iframe[@id='sp_message_iframe_204788']"))
time.sleep(2) # remove after testing
driver.find_element_by_xpath("//div[@class='message type-modal']//descendant::button[2]").click()

For reference on how to use XPath axes please refer this: https://www.lambdatest.com/blog/complete-guide-for-using-xpath-in-selenium-with-examples/#testid2.8.1

Please note webdriver_manager has some issues as of now, so use of chromedriver/firefox binary is highly suggested.(https://github.com/SergeyPirogov/webdriver_manager/issues/119)

Let me know if that works!

This does work, but using my code it doesn't - in order to capture information about web requests I have been using the selenium-wire library (from seleniumwire import webdriver) and starting the driver as driver = webdriver.Chrome(ChromeDriverManager().install(), seleniumwire_options=options, chrome_options=chrome_options) I wonder if my problem is actually related to selenium-wire. I kinda have to use selenium-wire, because I need to access the underlying requests. https://github.com/wkeeling/selenium-wire I'll update by opening post and add the full code. — qoob, Aug 15 '20 at 20:07
This led me to the correct solution, which was figuring out the issue lies within chromedriver when using selenium-wire. — qoob, Aug 16 '20 at 14:50

qoob · Answer 2 · 2020-08-16T14:58:56.760

Akshays answer led me to the right way - the issue is probably a bug in the selenium-wire library when using ChromeDriver (or atleast the current version 84.0.4147). If users have issues with the regular selenium library, Akshays answer is the correct one. The issue with selenium-wire can be fixed by using Firefox driver instead (current version of geckodriver v0.27.0):

from seleniumwire import webdriver
from selenium.webdriver.chrome.options import Options as chromeOptions
from selenium.webdriver.firefox.options import Options as firefoxOptions
from webdriver_manager.chrome import ChromeDriverManager
from webdriver_manager.firefox import GeckoDriverManager

if browser == 'chrome':
    chrome_options = chromeOptions()
    chrome_options.add_argument("--start-maximized")
    chrome_options.add_argument('--auto-open-devtools-for-tabs')
    chrome_options.add_argument('--log-level=2')
    driver = webdriver.Chrome(ChromeDriverManager().install(), seleniumwire_options=options, chrome_options=chrome_options)
elif browser == 'firefox':
    firefox_options = firefoxOptions()
    firefox_options.headless = True
    driver = webdriver.Firefox(executable_path=GeckoDriverManager().install(), seleniumwire_options=options, options=firefox_options)
    driver.maximize_window()

EDIT: Akshay's comment is actually even better - by removing --auto-open-devtools-for-tabs argument, switching to an iframe works even with chrome driver on selenium-wire webdriver.

Firefox is not without issues and for example using WebdriverWait would not work with custom wait times etc (only seems to work with 45 seconds as described here https://stackoverflow.com/a/27238388/13952179).

Glad that worked, somehow it also works on Chrome by ignoring this chrome options: # chrome_options.add_argument("--auto-open-devtools-for-tabs"). You might need to investigate on why so. Please up-vote and accept my previous answer if your query is solved. Thanks! — Akshay, Aug 16 '20 at 10:07
Thanks, that's a great catch! I'd rather use Chrome without the devtools option then since I'm running it headless anyway. — qoob, Aug 16 '20 at 14:54

Cant interact with iframe after switching to it Python/Selenium-wire

2 Answers2