1

Here is my settings:

CONCURRENT_REQUESTS = 64
CONCURRENT_REQUESTS_PER_DOMAIN = 100

I want to download requests and process response by spider in parallel way. But it doesn't work. Because it is sent to the Item Pipeline which processes it through several components that are executed sequentially!

ruibin
  • 113
  • 9
  • That is usually not a problem, because network latency is usually much much worse than the CPU time required by pipelines. Or callbacks, for that matter. If you are doing CPU-heavy stuff in pipelines, consider using Twisted’s `deferToThread` to run CPU-heavy stuff in a different thread. – Gallaecio Apr 17 '20 at 17:29

0 Answers0