According to FAQ, Scrapy explores depth first but I noticed it will process 10 to 30 requests from start_urls before running any deeper requests.
Can I change a parameter and force scrapy to explore depth before processing start_urls ?
Reference http://doc.scrapy.org/en/latest/faq.html#does-scrapy-crawl-in-breadth-first-or-depth-first-order
As Apalala suggested, if you really want to enforce a specific request order using Scrapy, inline requests are the way to go.
By design, Scrapy is not meant to follow a strict order for requests, and for most use cases Request.meta, priorities (from the Request constructor) or even the DEPTH_PRIORITY setting you mention are enough.
