scrapy-playwright: Why "waiting for selector to be visible" error is showing?

Question

I am trying to automatically select values using playwright. But it is not selecting the values.

waiting for selector "(//option[@value='2000000'])[2]" to be visible

this error message is showing. What is the reason behind it?

def start_requests(self):
        yield scrapy.Request(
            url="https://www.rightmove.co.uk/", 
            callback = self.parse, 
            meta= dict(
                    playwright = True,
                    playwright_include_page = True,
                    playwright_page_coroutines = [
                    # setting location
                    PageCoroutine("fill", "//input[@name='typeAheadInputField']", 'london'),
                    # clicking 'For Rent' button
                    PageCoroutine("click", selector = "//button[@class='ksc_button large primary searchPanelControls '][1]"),
                    # selecting price range
                    PageCoroutine("wait_for_timeout", 3000),
                    PageCoroutine("wait_for_selector", "(//option[@value='2000000'])[2]"),
                    PageCoroutine("select_option", "(//select/option[@value='2000000'])[2]"),
                    
                    # selecting bedroom range
                    PageCoroutine("wait_for_timeout", 3000),
                    PageCoroutine("wait_for_selector", "(//option[@value='5'])[1]"),
                    PageCoroutine("select_option", "(//select/option[@value='5'])[1]"),
                    # checking tick box
                    PageCoroutine("check", "//span[@class='tickbox--indicator']"),
                    PageCoroutine("wait_for_selector",  "//span[@class='tickbox--indicator']"),
                    PageCoroutine("wait_for_timeout", 3000),
                    # clicking search button
                    PageCoroutine("click", "//button[@class='//button[@id='submit"),
                    PageCoroutine("wait_for_selector", "//button[@class='//button[@id='submit"),
                    PageCoroutine("wait_for_timeout", 3000)
                    
                            ]
                        )
        )
        return super().start_requests()

output

playwright._impl._api_types.TimeoutError: Timeout 30000ms exceeded.
=========================== logs ===========================
waiting for selector "(//option[@value='2000000'])[2]" to be visible
  selector resolved to hidden <option value="2000000">2,000,000</option>

Answer 1

It is hard to say why an E2E script fails without knowing exactly what is the target page, but watching the output it seems like the problem is clear.

You are trying to target an element that is on the page, but is currently hidden (not visibile). Try to investigate on the reason why this is happening.

Answer 2

As said before, you're trying to select an element not visible. I tried to follow your scraper, if i look at the page "To Rent" for London, there's no option 2000000 in the price range menu. Try to set to an existing value (40000) and see if it works.