I've been trying to figure out how to webscrape this page: sick.com I can't figure it out. I've been trying Visual Web Ripper but it doesn't pass the submit form, because it doesn't remember the cookie. Do you have any ideas? Sick.com is ok with me extracting the data, but they are lazy to give us the database.
Asked
Active
Viewed 194 times
1 Answers
0
I'm not too familiar with Visual Web Ripper, but it sounds like you simply need to select an alternative tool that will track the cookies for you. There are a lot of open source tools like scrapy and Mechanize. I used screen-scraper and it's tracking the cookies just fine.
That said, you might also ensure that when using Visual Web Ripper you're requesting the initial search page before requesting the search results. That is, an initial request should set the cookies, which will then propagate when making subsequent requests.
todd
- 79
- 2