0

I have a use case like this. Suppose if I crawl a website abc.com using scrapy playwright the page it loads are of 3 different types of pages like

page1->#selector1 
page2->#selector2
page3->#selector3

and it changes dynamically there is no guarantee which loads first. I want to click on the selector based on which is loaded using scrapy-playwright.

Is there a way to do this dynamically based selector loaded?

I tried using

PageMethod("click","#selector1"),
PageMethod("click","#selector2"),
PageMethod("click","#selector3")

and this is getting timed out because at that point in time #selector2 is loaded first on the page causing the operation to fail as the first method is PageMethod("click","#selector1")

User10007
  • 1
  • 1

1 Answers1

0

Maybe you can check the selector is visible and then you can click it?

if PageMethod("isVisible","#selector1"):
    PageMethod("click","#selector1")    
if PageMethod("isVisible","#selector2"):
    PageMethod("click","#selector2")    
if PageMethod("isVisible","#selector3"):
    PageMethod("click","#selector3")

I think you can usue the method isVisible, because scrapy_playwright support the same you have in playwright: https://github.com/scrapy-plugins/scrapy-playwright#supported-methods

And of course, isVisible() is a supported method in playwrgiht: https://playwright.dev/python/docs/api/class-page#page-is-visible

Jaky Ruby
  • 1,409
  • 1
  • 7
  • 12
  • Thanks for reply how do I add condition checks inside an array beacuse scrapy accepts pagemethods like this in array what would the syntax be? `playwright_page_methods=[ PageMethod("wait_for_selector", "div.quote"), PageMethod("evaluate", "window.scrollBy(0, document.body.scrollHeight)"), PageMethod("wait_for_selector", "div.quote:nth-child(11)"), # 10 per page ],` – User10007 Nov 10 '22 at 03:32
  • I am not very familiar with scrapy_playwright, but sorry I do not understand the question of this comment, it does not look similar to the main question of the post – Jaky Ruby Nov 10 '22 at 18:06