How to scrape all reviews from walmart review page (ex:http://www.walmart.com/ip/Ematic-9-Dual-Screen-Portable-DVD-Player-with-Dual-DVD-Players-ED929D/28806789) if they are on different pages?I scrape by mechanize(nokogiri) but it can't click on button(it is not part of form,then I can't submit it)
<button class="paginator-btn paginator-btn-next"><span
class="visuallyhidden">Next Page</span></button>
and I can't go to next page.How to solve this problem?
I think it may be easier than that. If you pay attention to the product url, you see that there is some kind of ID at the end of the url:
http://www.walmart.com/ip/Ematic-9-Dual-Screen-Portable-DVD-Player-with-Dual-DVD-Players-ED929D/28806789
If you get that ID, you could take the reviews root page (https://www.walmart.com/reviews/product/) and concat the ID of the product:
Now, you can iterate over the products, take the trailing ID, and go to each reviews page to get all the reviews.
Hope it helped.
The page you posted is empty for me. However, what I see is that the element is a button, therefore, what you need to do is look for the form and then submit it.
Example taken from Clicking a button with Ruby mechanize (in case the link stops working for some reason):
# get the form
form = agent.page.form_with(:name => "my-form")
# get the button you want from the form
button = form.button_with(:value => "Search")
# submit the form using that button
agent.submit(form, button)
Credit to @flaviu and @serabe from the question stated.
To do the scraping, you should save the root url and go to the review pages, get the reviews, go back to the root url, and so on.
