2

This scraper works fine. I only want to get the titles of the items on this page.

walmart scraper

In scrapy shell, using the view(response) function reveals a web page that says "Your web browser is not accepting cookies." Even when I add USER_AGENT information to a scrapy shell launch.

"Your web browser is not accepting cookies."

As a result, the scraper doesn't manage to scrape any information. Things that I have changed:

COOKIES_ENABLED = True
COOKIES_DEBUG = True
ROBOTSTXT_OBEY = False
USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36'
DOWNLOADER_MIDDLEWARES = {'walmartscraper.middlewares.WalmartscraperDownloaderMiddleware': 543,}

I have a feeling I need to add/change something in the middlewares section (it is still the default code) and/or implement requests somewhere. This is the first time I've worked with cookies while scraping and the information I've found hasn't helped me figure this out.

Any advice is very much appreciated. Thank you.

jayjey
  • 127
  • 9
  • Does this answer your question? [What should i do to enable cookies and use scrapy for this url?](https://stackoverflow.com/questions/44558997/what-should-i-do-to-enable-cookies-and-use-scrapy-for-this-url) – Leonardo Kuffo Aug 30 '20 at 05:43

0 Answers0