how to scrape dynamic web pages using ui path studio

Question

I tried web scraping reviews from a website called https://mmb.moneycontrol.com and the reviews posted on this website are loaded dynamically. The amount of reviews scraped are 40 only i tried changing the "MaxNoOfResults" variable to higher value (300,400...) but the amount of scraped reviews remained the same. I even tried assigning a new excel sheet every time I scraped even that did not fetch any different results can someone please help me increase the scraped reviews

Answer 1

The problem is that the reviews are loaded upon scroll down it seems. The UiPath process is only able to "see" the elements that have been loaded.

I would suggest to use the Send Hotkey activity with "Page Down" key to scroll down the page and force the loading of the elements before doing the scraping.

The challenge is going to be to decide when to stop, so you will need to decide on some logic for that (there is no obvious or generic approach to this I am afraid)