0

I have developed a simple scrapy project to crawl a website. The crawler works fine on my local machine but when I try to deploy it to Scrapy cloud provided by scrapinghub.com the spider shows 0 pages crawled and after 180 sec (default timeout) it just stops. What can be the issue with it? I can try increasing the timeout but I think 180 secs. is a sufficient long time for the crawler to get response from the website.

shubham003
  • 703
  • 2
  • 9
  • 20
  • Anything in the logs? – Wiggy A. Dec 04 '17 at 12:25
  • Not enough detail in question to answer. Include logs, config and/or any errors. – Amit Tripathi Dec 04 '17 at 12:34
  • 2
    There may be various reasons. E.g. it's possible that the spider uses different settings when running locally and on Scrapy Cloud, or maybe IP address of the crawling worker on Scrapy Cloud is being blocked by the target website. Please try adding more info to the question. – starrify Dec 04 '17 at 12:35

0 Answers0