-2

I am building a scraper to scrape content using guzzle and symfony dom crawler But I run into an issue.

The page I am scraping has multiple Iframe servers They default iframe is shown when the scraper loads the page but in order to get the other servers it needs to click there buttons and so it reflects the server iframe.

How do I do that?

DisappointedByUnaccountableMod
  • 6,656
  • 4
  • 18
  • 22
Zack Snyder
  • 61
  • 8
  • When building a web scraping tool, you need to emulate a browser. In this case, clicking those buttons does something, and you need to emulate it. Maybe they set a hidden field, maybe they perform a XHR/fetch, maybe the present a CAPTCHA. Whatever they do, it will end up as an HTTP request in some fashion, and that's what you need to emulate. Pull up your developer tools and watch what goes across the wire. – Chris Haas Nov 29 '21 at 14:02

1 Answers1

0

Guzzle can't render Javascript websites so it doesn't support that

There are two ways around this either check for network tabs and send the request to the ajax link they get the iframes from, or use symfony Panther as headless browser library they support javascript websites and even Bypass Cloudflare Protection

Zack Snyder
  • 61
  • 8