1

I want to use Crawlera only for some requests in a Scrapy spider. So I want to set CRAWLERA_ENABLED differently for different requests. Is it possible?

Aminah Nuraini
  • 18,120
  • 8
  • 90
  • 108

1 Answers1

1

You can use the dont_proxy key in meta for those requests you don't want to use Crawlera. E.g.

# Supposing you have crawlera enabled in `settings.py`
yield scrapy.Request(
    url, 
    meta={"dont_proxy": True}, 
    callback=self.parse
)
Thiago Curvelo
  • 3,711
  • 1
  • 22
  • 38
  • Once I use `dont_proxy`, it won't set Crawlera when I retry a request – Aminah Nuraini Oct 26 '18 at 08:38
  • 1
    `scrapy-crawlera` checks if the key exists in `meta` [(link to the code)](https://github.com/scrapy-plugins/scrapy-crawlera/blob/13a893f940254629a6bb2ef7d70ed887c240278b/scrapy_crawlera/middleware.py#L177). Try to remove `dont_proxy` from `meta` instead of setting it into `False`. – Thiago Curvelo Nov 05 '18 at 19:02
  • Yup, that's what happened when I check `scrapy-crawlera`'s code. – Aminah Nuraini Nov 07 '18 at 05:58