-4

Suppose you have a play store comments page in which hundreds of people commenting regularly. My idea is to take data from the play store comments page not only on the same page but also when you're scrolling. So it should take All the data from the comment pages Who has commented on it? If he has given any ratings or not. Including his/her photos.

How do I approach this problem and what tools I can use for this? Share all the links where I can read more about this type of problem statement.

Srikant Singh
  • 149
  • 1
  • 5
  • 17

1 Answers1

0

I would use scrapy with looping requests to the ajax URL. Logic to terminate the loop can handled a few ways depending on what data is available.

  1. If each list of responses is expected to be N comments long and you receive a response with less than N comments, break
  2. If there is a total comments value available, page through until the number of pages requested is equal to the total number of comments divided by the number of comments per page.
  3. Other solutions exists

How to scrape all contents from infinite scroll website? scrapy

Scraping Infinite Scrolling Pages with "load more" button using Scrapy

https://stackoverflow.com/search?q=scrapy+infinite+scroll

pwinz
  • 303
  • 2
  • 14